multi-modal AI workloads foundations

Signed-off-by: Max Pumperla <max.pumperla@googlemail.com>

Author: maxpumperla

courses/.gitignore

@@ -12,4 +12,4 @@ public/
 
 # Content production folder
 production/
-outputs/
+outputs/

courses/foundations/Multimodal AI Workloads/00_overview/README.ipynb

@@ -0,0 +1,111 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Multi-modal AI pipeline\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "<div align=\"left\">\n",
+    "<a target=\"_blank\" href=\"https://console.anyscale.com/\"><img src=\"https://img.shields.io/badge/🚀 Run_on-Anyscale-9hf\"></a>&nbsp;\n",
+    "<a href=\"https://github.com/anyscale/multimodal-ai\" role=\"button\"><img src=\"https://img.shields.io/static/v1?label=&amp;message=View%20On%20GitHub&amp;color=586069&amp;logo=github&amp;labelColor=2f363d\"></a>&nbsp;\n",
+    "</div>\n",
+    "\n",
+    "💻 Run this entire tutorial on [Anyscale](https://www.anyscale.com/) for free:\n",
+    "**https://console.anyscale.com/template-preview/image-search-and-classification** or access the repo [here](https://github.com/ray-project/ray/tree/master/doc/source/ray-overview/examples/e2e-multimodal-ai-workloads).\n",
+    "\n",
+    "This tutorial focuses on the fundamental challenges of multimodal AI workloads at scale:\n",
+    "\n",
+    "- **🔋 Compute**: managing heterogeneous clusters, reducing idle time, and handling complex dependencies\n",
+    "- **📈 Scale**: integrating with the Python ecosystem, improving observability, and enabling effective debugging\n",
+    "- **🛡️ Reliability**: ensuring fault tolerance, leveraging checkpointing, and supporting job resumability\n",
+    "- **🚀 Production**: bridging dev-to-prod gaps, enabling fast iteration, maintaining zero downtime, and meeting SLAs\n",
+    "\n",
+    "This tutorial covers how Ray addresses each of these challenges and shows the solutions hands-on by implementing scalable batch inference, distributed training, and online serving workloads.\n",
+    "\n",
+    "- [**`01-Batch-Inference.ipynb`**](https://github.com/anyscale/multimodal-ai/tree/main/notebooks/01-Batch-Inference.ipynb): ingest and preprocess data at scale using [Ray Data](https://docs.ray.io/en/latest/data/data.html) to generate embeddings for an image dataset of different dog breeds and store them.\n",
+    "- [**`02-Distributed-Training.ipynb`**](https://github.com/anyscale/multimodal-ai/tree/main/notebooks/02-Distributed-Training.ipynb): preprocess data to train an image classifier using [Ray Train](https://docs.ray.io/en/latest/train/train.html) and save model artifacts to a model registry (MLOps).\n",
+    "- [**`03-Online-Serving.ipynb`**](https://github.com/anyscale/multimodal-ai/tree/main/notebooks/03-Online-Serving.ipynb): deploy an online service using [Ray Serve](https://docs.ray.io/en/latest/serve/index.html), that uses the trained model to generate predictions.\n",
+    "- Create production batch [**Jobs**](https://docs.anyscale.com/platform/jobs/) for offline workloads like embedding generation, model training, etc., and production online [**Services**](https://docs.anyscale.com/platform/services/) that can scale.\n",
+    "\n",
+    "<img src=\"https://raw.githubusercontent.com/anyscale/multimodal-ai/refs/heads/main/images/overview.png\" width=1000>\n",
+    "\n",
+    "## Development\n",
+    "\n",
+    "The application is developed on [Anyscale Workspaces](https://docs.anyscale.com/platform/workspaces/), which enables development without worrying about infrastructure—just like working on a laptop. Workspaces come with:\n",
+    "- **Development tools**: Spin up a remote session from your local IDE (Cursor, VS Code, etc.) and start coding, using the same tools you love but with the power of Anyscale's compute.\n",
+    "- **Dependencies**: Install dependencies using familiar tools like pip or uv. Anyscale propagates all dependencies to the cluster's worker nodes.\n",
+    "- **Compute**: Leverage any reserved instance capacity, spot instance from any compute provider of your choice by deploying Anyscale into your account. Alternatively, you can use the Anyscale cloud for a full serverless experience.\n",
+    "  - Under the hood, a cluster spins up and is efficiently managed by Anyscale.\n",
+    "- **Debugging**: Leverage a [distributed debugger](https://docs.anyscale.com/platform/workspaces/workspaces-debugging/#distributed-debugger) to get the same VS Code-like debugging experience.\n",
+    "\n",
+    "Learn more about Anyscale Workspaces in the [official documentation](https://docs.anyscale.com/platform/workspaces/).\n",
+    "\n",
+    "<div align=\"center\">\n",
+    "  <img src=\"https://raw.githubusercontent.com/anyscale/multimodal-ai/refs/heads/main/images/compute.png\" width=600>\n",
+    "</div>\n",
+    "\n",
+    "### Additional dependencies\n",
+    "\n",
+    "You can choose to manage the additional dependencies via `uv` or `pip`. \n",
+    "\n",
+    "```bash\n",
+    "# UV setup instructions\n",
+    "uv init .  # this creates pyproject.toml, uv lockfile, etc.\n",
+    "ray_wheel_url=http://localhost:9478/ray/$(pip freeze | grep -oP '^ray @ file:///home/ray/\\.whl/\\K.*')\n",
+    "uv add \"$ray_wheel_url[data, train, tune, serve]\"  # to use anyscale's performant ray runtime\n",
+    "uv add $(grep -v '^\\s*#' requirements.txt)\n",
+    "uv add --editable ./doggos\n",
+    "```\n",
+    "\n",
+    "```bash\n",
+    "# Pip setup instructions\n",
+    "pip install -q -r /home/ray/default/requirements.txt\n",
+    "pip install -e ./doggos\n",
+    "```\n",
+    "\n",
+    "**Note**: Run the entire tutorial for free on [Anyscale](https://console.anyscale.com/)—all dependencies come pre-installed, and compute autoscales automatically. To run it elsewhere, install the dependencies from the [`containerfile`](https://github.com/anyscale/multimodal-ai/tree/main/containerfile) and provision the appropriate GPU resources.\n",
+    "\n",
+    "## Production\n",
+    "Seamlessly integrate with your existing CI/CD pipelines by leveraging the Anyscale [CLI](https://docs.anyscale.com/reference/quickstart-cli) or [SDK](https://docs.anyscale.com/reference/quickstart-sdk) to deploy [highly available services](https://docs.anyscale.com/platform/services) and run [reliable batch jobs](https://docs.anyscale.com/platform/jobs). Developing in an environment nearly identical to production—a multi-node cluster—drastically accelerates the dev-to-prod transition. This tutorial also introduces proprietary RayTurbo features that optimize workloads for performance, fault tolerance, scale, and observability.\n",
+    "\n",
+    "```bash\n",
+    "anyscale job submit -f /home/ray/default/configs/generate_embeddings.yaml\n",
+    "anyscale job submit -f /home/ray/default/configs/train_model.yaml\n",
+    "anyscale service deploy -f /home/ray/default/configs/service.yaml\n",
+    "```\n",
+    "\n",
+    "## No infrastructure headaches\n",
+    "Abstract away infrastructure from your ML/AI developers so they can focus on their core ML development. You can additionally better manage compute resources and costs with [enterprise governance and observability](https://www.anyscale.com/blog/enterprise-governance-observability) and [admin capabilities](https://docs.anyscale.com/administration/overview) so you can set [resource quotas](https://docs.anyscale.com/reference/resource-quotas/), set [priorities for different workloads](https://docs.anyscale.com/administration/cloud-deployment/global-resource-scheduler) and gain [observability of your utilization across your entire compute fleet](https://docs.anyscale.com/administration/resource-management/telescope-dashboard).\n",
+    "Users running on a Kubernetes cloud (EKS, GKE, etc.) can still access the proprietary RayTurbo optimizations demonstrated in this tutorial by deploying the [Anyscale Kubernetes Operator](https://docs.anyscale.com/administration/cloud-deployment/kubernetes/)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "```{toctree}\n",
+    ":hidden:\n",
+    "\n",
+    "notebooks/01-Batch-Inference\n",
+    "notebooks/02-Distributed-Training\n",
+    "notebooks/03-Online-Serving\n",
+    "```"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}