Readme updates (#69)

Signed-off-by: Claudia <[email protected]>

Author: cmisale

HEALTH_CHECKS.md

@@ -35,6 +35,8 @@ Here is a breakdown of the existing health checks:
 
 These checks are configured to run periodically (e.g., hourly), and results are accessible via Prometheus, direct API queries or labels on the worker nodes.
 
+![image](figures/periodic-check-flow.svg)
+
 ## Deep Diagnostics and Node Labeling
 
 Autopilot's periodic health checks, will label the worker nodes according to the result obtained.
@@ -55,11 +57,13 @@ autopilot.ibm.com/gpuhealth: WARN
 ### Invasive health checks
 
 The invasive DCGM diagnostics level 3 health check, executed automatically only on nodes that have free GPUs. This deeper analysis is needed to reveal problems in the GPUs that can be found only after running level 3 DCGM diagnostic.
+
+![image](figures/invasive-check-flow.svg)
+
 This type of diagnostics can help deciding if the worker node should be used for running workloads or not. To facilitate this task, Autopilot will label nodes with key `autopilot.ibm.com/dcgm.level.3`.
 
 If a fatal error is found, the `gpuhealth` label is updated to evict.
 
-
 ```yaml
 autopilot.ibm.com/gpuhealth: EVICT
 ```

README.md

@@ -1,16 +1,14 @@
 # AI Training Autopilot
 
-[[README refresh ongoing]]
-
 Autopilot is a Kubernetes-native daemon that continuously monitors and evaluates GPUs, network and storage health, designed to detect and report infrastructure-level issues during the lifetime of AI workloads. It is an open-source project developed by IBM Research.
 
 In AI training jobs, which may run for weeks or months, anomalies in the GPUs and network can happen anytime and often go undetected. In this case, performance degrades suddenly and a deep diagnostic is needed to identify the root cause, delaying or deleting the current job. Similarly, hardware anomalies can greatly disrupt the throughput and latency of an AI inference server.
 
-The role of Autopilot is to detect and report any problems that are detected during the lifetime of the job and the existence of a cluster.
+The role of Autopilot is to detect and report any problems that are detected by its health checks during the lifetime of the job and the existence of a cluster.
 
 It implements a set of health checks evaluating the status of the system. These health checks focus mainly on subtle/software issues (i.e., row-remapping or PCIe link degradation), but also run connectivity tests (i.e., ping, iperf) to verify that secondary NICs are reachable. It can also verify that persistent volume claims (PVC) creation is functional for a given storage class.
 
-![image](https://media.github.ibm.com/user/96687/files/0d466863-a19e-459d-a492-e2275377d4b9)
+![image](figures/autopilot-daemon-pod.png)
 
 Autopilot is deployed as a Kubernetes DaemonSet on all worker nodes that have GPUs. Each pod exposes a Service that can be accessed through RESTful API to request the execution of health checks. Therefore, each health check has its own entry point, but also a generic “status” entry point is provided.
 
@@ -20,25 +18,7 @@ The main code is written in Go, while health checks are written in a combination
 
 If Autopilot requires full access to GPUs to run more invasive workloads, it will spawn a separate job with resources requests and limits set.
 
-![image](https://media.github.ibm.com/user/96687/files/4a7c81ba-857a-43d4-bc82-0784ef81b270)
-
-The toolkit currently provides health checks for pre-flight and post-flight phases, while in-flight checks will be enabled in the future. In more details (list subject to change):
-
-- pre-flight checks
-
-  - validate infrastructure before the start of jobs
-
-- in-flight checks
-
-  - workload and system performance is continuously monitored
-
-  - detect anomaly, and issue notification
-
-  - controllers can take actions if errors are found
-
-- post-flight checks
-
-  - validate infrastructure once the job ends
+![image](figures/autopilot-main-loop.svg)
 
 ## Health Checks
 
@@ -62,6 +42,15 @@ Results from health checks are exported as Prometheus Gauges, so that users and
 
 Detailed description of all the health checks, can be found in [HEALTH_CHECKS.md](HEALTH_CHECKS.md).
 
+### Diagnostics and Node Labeling
+
+Autopilot's periodic and invasive health checks, will label the worker nodes according to the result obtained.
+Lightweight and invasive health checks, may use different labeling system. Refer to [HEALTH_CHECKS.md](HEALTH_CHECKS.md) for more details about the labels format.
+
+The information saved in the labels, can be used by admins, kube-scheduler or other workload management systems like [CodeFlare](https://project-codeflare.github.io/appwrapper/arch-fault-tolerance/) to steer the execution of workloads for enhanced fault tolerance.
+
+![image](figures/big-picture.svg)
+
 ## Install
 
 To learn how to install Autopilot, please refer to [SETUP.md](SETUP.md)