paper.tex

\documentclass[10pt,twocolumn]{article}

% Line spacing
\renewcommand{\baselinestretch}{1.05}
\widowpenalty=10000
\clubpenalty=10000
\hyphenpenalty=300
\tolerance=1000

% Essential packages
\usepackage[T1]{fontenc}
\usepackage{mathpazo}
\usepackage[scaled=0.9]{helvet}
\usepackage{courier}
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\newtheorem{definition}{Definition}
\usepackage{booktabs}
\usepackage{tabularx}
\usepackage{xcolor}
\usepackage{listings}
\PassOptionsToPackage{hyphens,spaces,obeyspaces}{url}
\usepackage[round,authoryear]{natbib}
\bibliographystyle{plainnat}
\usepackage{hyperref}
\usepackage[nameinlink]{cleveref}
\usepackage{subcaption}
\usepackage{enumitem}
\usepackage{titlesec}
\usepackage{fancyhdr}
\usepackage{tikz}
\usetikzlibrary{shapes,arrows,positioning,shadows,calc,backgrounds,decorations.pathreplacing,fit}

% Caption styling
\captionsetup{
  font=small,
  labelfont=bf,
  labelsep=period,
  justification=justified,
  singlelinecheck=false,
  skip=8pt
}
\captionsetup[table]{position=top}
\captionsetup[figure]{position=bottom}
\captionsetup[subfigure]{justification=centering}

% Section spacing
\titlespacing*{\section}{0pt}{0.8\baselineskip}{0.5\baselineskip}
\titlespacing*{\subsection}{0pt}{0.6\baselineskip}{0.4\baselineskip}
\titlespacing*{\subsubsection}{0pt}{0.5\baselineskip}{0.3\baselineskip}

% Page geometry
\usepackage[
  letterpaper,
  top=0.75in,
  bottom=1in,
  left=0.75in,
  right=0.75in,
  columnsep=0.25in
]{geometry}

% Header
\pagestyle{fancy}
\fancyhf{}
\renewcommand{\headrulewidth}{0pt}
\fancyfoot[C]{\fontsize{9}{11}\selectfont\thepage}
\setlength{\headheight}{12pt}
\setlength{\headsep}{18pt}

\fancypagestyle{plain}{%
  \fancyhf{}%
  \renewcommand{\headrulewidth}{0pt}%
  \fancyhead{}%
  \fancyfoot{}%
}

% Code styling
\definecolor{codegreen}{rgb}{0,0.5,0}
\definecolor{codegray}{rgb}{0.4,0.4,0.4}
\definecolor{codepurple}{rgb}{0.58,0,0.82}
\definecolor{backcolour}{rgb}{0.98,0.98,0.98}
\definecolor{accentblue}{RGB}{33,150,243}
\definecolor{accentorange}{RGB}{255,152,0}
\definecolor{accentgreen}{RGB}{76,175,80}

\lstdefinestyle{pythonstyle}{
    backgroundcolor=\color{backcolour},
    commentstyle=\color{codegreen},
    keywordstyle=\color{blue},
    stringstyle=\color{codepurple},
    basicstyle=\ttfamily\scriptsize,
    breaklines=true,
    captionpos=b,
    keepspaces=true,
    numbers=left,
    numbersep=4pt,
    numberstyle=\tiny\color{codegray},
    showstringspaces=false,
    tabsize=2,
    language=Python,
    frame=single,
    framerule=0.5pt,
    rulecolor=\color{black!20},
    xleftmargin=3pt,
    xrightmargin=3pt,
    aboveskip=5pt,
    belowskip=5pt
}
\lstset{style=pythonstyle}

% Hyperref
\hypersetup{
    colorlinks=true,
    allcolors=blue,
    pdftitle={MLPerf EDU: A Pedagogical Benchmarking Suite for ML Systems Education},
    pdfauthor={Vijay Janapa Reddi}
}

% Title formatting
\usepackage{titling}
\pretitle{\begin{center}\vspace*{-1em}}
\posttitle{\vspace{0.8em}\end{center}}

\title{
  \Huge\bfseries MLPerf EDU\\[0.4em]
  \Large\normalfont A Pedagogical Evaluation Framework\\for AI Systems Benchmarking
}

\author{
  \fontsize{12}{15}\selectfont
  Vijay Janapa Reddi\\[0.2em]
  \fontsize{11}{14}\selectfont
  Harvard University
}

\date{\href{https://github.com/harvard-edge/mlperf-edu}{github.com/harvard-edge/mlperf-edu}}

\begin{document}

\thispagestyle{plain}
\maketitle
\begingroup
\renewcommand{\thefootnote}{}%
\footnotetext{Part of the \emph{Machine Learning Systems} book ecosystem; context: \url{https://mlsysbook.ai/}.}%
\endgroup

\begin{abstract}
MLPerf is the industry standard for measuring ML systems performance, but its enterprise-scale requirements---large accelerator clusters, large datasets, and production submission harnesses---make it difficult to use in coursework and many academic artifact evaluations. We present \textbf{MLPerf EDU}, an executable, pedagogical benchmark suite that keeps the central MLPerf ideas---fixed workloads, scenarios, quality targets, provenance, and auditable reports---while making the default path runnable on laptop-class hardware. The current implementation exposes a single \texttt{mlperf} command with \texttt{min}, \texttt{max}, and \texttt{pro} profiles; suite/profile/workload/variant selection; automatic fetch and audit flows; and default JSON, HTML, CSV, and \texttt{.provd.json} artifacts. The registry contains 30 workload rows across language, small-language-model serving, vision, recommendation, tiny, agent, distributed, graph, time-series, and reinforcement-learning suites. Five workloads are currently score-bearing, four are performance-bearing, and the rest are systems-only research or teaching scaffolds. Public-result candidates carry structured dataset/model dossiers, reference-run protocols, checkpoint lineage for training-to-inference flows, and generated review packets for MLCommons-style feedback. The goal is not to replace official competitive MLPerf submissions, but to provide a SPEC-like educational and research benchmark that students and researchers can actually run, inspect, modify, and cite.
\end{abstract}

% ============================================================================
\section{Introduction}
\label{sec:intro}
% ============================================================================

The rapid growth of machine learning has driven a parallel expansion in ML hardware, compilation stacks, and deployment frameworks. To impose order on this ecosystem, the community coalesced around MLPerf~\citep{mattson2020}, establishing the definitive standard for measuring ML systems performance. MLPerf mandates strict rules: fixed reference implementations, rigid dataset constraints, mathematically verified quality targets, and standardized measurement scenarios.

However, the very rigor that makes MLPerf the gold standard renders it inaccessible to educational settings. A single MLPerf Training run requires enterprise-grade hardware (e.g., 8$\times$ H100 nodes), terabytes of storage, and containerized deployment infrastructure. Students cannot replicate these conditions on their laptops. As a result, instructors rely on ad-hoc assignments that lack the architectural diversity, measurement discipline, and reproducibility guarantees that define professional benchmarking.

\textbf{Why now.} Three converging trends make this gap untenable. First, systems-aware AI techniques (quantization, pruning, KV-caching, speculative decoding) require students to learn to \emph{evaluate} optimizations, not merely implement models. Second, agentic AI introduces novel performance dimensions---retrieval latency, tool-call overhead, multi-step cost---that no existing educational benchmark addresses. Third, the industry needs graduates who understand benchmarking culture: the discipline of fair comparison, reproducible measurement, and submission-grade reporting.

This paper presents \textbf{MLPerf EDU}, a pedagogical evaluation framework that teaches students \emph{how to evaluate ML systems}, not just how to build them. Unlike ad-hoc assignments or simplified tutorials, MLPerf EDU provides a formal evaluation methodology grounded in MLPerf conventions, enabling students to reason about systems performance the way industry practitioners do. The current executable registry contains 30 workload rows. The public-result slice contains five score-bearing workloads with quality targets and four performance-bearing workloads with functional checks; the remaining rows expose optimization, systems, and research variants for classroom and academic studies. \Cref{fig:architecture} shows the system architecture.

\textbf{Formal Evaluation Abstraction.} Every MLPerf EDU benchmark instantiates a common evaluation tuple:

\begin{definition}[Evaluation Tuple]
A benchmark configuration $\mathcal{S}$ = (Model, Data, Hardware, Scenario, Constraints) \\
produces a measurement vector $\mathcal{M}$ = (Latency, Throughput, Accuracy, Energy). The \emph{Constraints} component captures the profile, quality target, run budget, and public-result policy.
\end{definition}
\vspace{-3pt}
Students learn to reason across these dimensions by varying one component and observing its effect on $\mathcal{M}$. For example, fixing $\mathcal{S}$ for NanoGPT and changing only \emph{Scenario} from SingleStream to Server (Poisson arrivals at 100 QPS) increases p99 latency from 7.84\,ms to $\sim$12\,ms due to queuing effects---a concrete lesson in Little's Law.

\textbf{Contributions.}
\begin{enumerate}[leftmargin=*, itemsep=2pt]
\item \textbf{Evaluation abstraction.} The evaluation tuple $\mathcal{S} \rightarrow \mathcal{M}$ (Definition 1) provides a formal, reusable framework for reasoning about ML systems performance. This abstraction---not the benchmarks themselves---is the primary intellectual contribution.
\item \textbf{Pedagogical evaluation framework.} A modular benchmarking architecture---dataset factory, load generator, power profiler, verifier, grader, and report generator---that teaches evaluation methodology. Students act as MLPerf ``submitters'' whose JSON, HTML, CSV, and provenance artifacts can be verified and ranked.
\item \textbf{Three-axis regime taxonomy.} Rather than label workloads by a single "bottleneck class,'' every workload is placed on an orthogonal three-axis grid: \emph{working set} (cache-resident vs.\ DRAM-bound, measured against the reference LLC), \emph{arithmetic intensity} (compute-bound vs.\ bandwidth-bound, measured against the reference roofline ridge), and \emph{dispatch} (device-saturated vs.\ dispatch-bound, measured as achieved fraction of peak). The axes are orthogonal because bumping batch size shifts dispatch without changing working set, bumping embedding-table size shifts working set without changing intensity, and switching prefill$\rightarrow$decode shifts intensity without changing working set. The current registry spans 30 workload rows across language, SLM serving, vision, recommendation, tiny, agent, distributed, graph, time-series, and reinforcement-learning suites.
\item \textbf{Public-result contract.} Score-bearing workloads declare explicit quality targets, verified baselines, reference-run protocols, and dataset release status. Performance-bearing workloads declare functional checks and, when inference depends on training, structured checkpoint lineage.
\item \textbf{Auditable artifacts.} Every run can emit JSON, HTML, CSV, and \texttt{.provd.json} artifacts with hardware fingerprints, dataset/model dossiers, source/license status, quality metadata, and checkpoint provenance.
\item \textbf{Review packets.} The repository can generate one-page Markdown packets for the nine current public-result candidates, giving MLCommons reviewers a compact view of commands, assets, quality targets, public warnings, and provenance.
\end{enumerate}

% ============================================================================
% Architecture Diagram
% ============================================================================

\begin{figure*}[t]
\centering
\includegraphics[width=0.95\textwidth]{figures/architecture.pdf}
\caption{\textbf{MLPerf EDU System Architecture.} Workloads flow through a unified CLI harness with asset fetch, audit, load generation, optional power profiling, report generation, and provenance verification. Students act as ``submitters'' who optimize reference implementations and produce verifiable JSON, HTML, CSV, and provenance artifacts. The bottom row shows the formal evaluation tuple: each workload instantiates $\mathcal{S} = (\text{Model, Data, Hardware, Scenario, Constraints})$ and produces metrics $\mathcal{M} = (\text{Latency, Throughput, Accuracy, Energy})$.}
\label{fig:architecture}
\end{figure*}

% ============================================================================
\section{Design Principles}
\label{sec:design}
% ============================================================================

Three principles guide MLPerf EDU's design, each addressing a specific failure mode of existing pedagogical tools.

\subsection{Pedagogical Transparency over Performance}

Production benchmarks optimize for throughput via custom CUDA kernels, mixed-precision arithmetic, and operator fusion---optimizations that make computation invisible to students. MLPerf EDU inverts this priority for its reference workloads: local models use inspectable Python/PyTorch with explicit operations and named dimensions, while the SLM suite uses permissive off-the-shelf Hugging Face models to expose modern serving behavior. A student reading the DS-CNN implementation sees depthwise and pointwise convolutions as separate \texttt{nn.Conv2d} layers with \texttt{groups=in\_channels}, not a fused kernel.

\subsection{Real Data with Provenance}

A benchmark without real data teaches stress testing, not systems evaluation. Students must observe how data distributions affect convergence. \Cref{tab:datasets} documents representative datasets, while the executable registry carries structured asset dossiers for each dataset and model. Data is either bundled for zero-network runs, fetched from public upstream sources, or marked with an explicit public-release warning when the release policy still needs owner or MLCommons review.

\subsection{Complete Train--Checkpoint--Inference Loop}

Many tools provide only training or only inference. MLPerf EDU provides the complete pipeline: training code produces checkpoints, checkpoints load into inference harnesses, and harnesses measure latency under realistic scenarios. The current registry records dataset, model, checkpoint, profile, quality, and public-result metadata in one place, ensuring deterministic, reproducible experiments.

\begin{lstlisting}[caption={\textbf{Registry-driven workload metadata.} Workload rows declare the user-facing selector, suite, profile coverage, assets, and quality policy.},label={lst:factory},float=t]
workload:
  id: nanogpt-train
  suite: language
  profiles: [min, max, pro]
  runner: nanogpt
  dataset: tinyshakespeare
  quality_target:
    metric: cross_entropy_loss
    direction: lower
    threshold: 2.3
    target_basis: reference_runs
  public_status: score-bearing
\end{lstlisting}

% ============================================================================
\section{Workload Suite}
\label{sec:suite}
% ============================================================================

The executable MLPerf EDU registry currently organizes 30 workload rows across ten suites: language, SLM serving, vision, recommendation, tiny, agent, distributed, graph, time-series, and reinforcement learning. The public CLI exposes only suite, profile, workload, and variant. \texttt{min} is the minimum representative path, \texttt{max} is the comparable full-suite path, and \texttt{pro} exposes research variants and optimization knobs. \Cref{tab:workloads} summarizes the representative workload families that motivated the initial registry; the native registry and generated review packets are the source of truth for the current executable rows.

\textbf{Regime classification.} Rather than assign each workload a single "bottleneck class,'' we place every workload on an orthogonal three-axis grid introduced in §\ref{sec:design}: \emph{working set} (cache-resident vs.\ DRAM-bound), \emph{arithmetic intensity} (compute-bound vs.\ bandwidth-bound), and \emph{dispatch} (device-saturated vs.\ dispatch-bound). Each axis is measured against published reference thresholds (Apple M1 base: 12\,MB LLC, 30 FLOP/byte ridge, 68\,GB/s unified memory). When a workload's measured value falls in the grey band between two axis thresholds, it is recorded as \texttt{unmeasured} rather than assigned a guess---the linter (\texttt{tools/check\_taxonomy.py}) rejects any schema violation where a categorical claim disagrees with the numerical evidence. This honesty has practical consequences: fifteen of our initial static-analysis classifications did not survive measurement and were revised, and one cell of the $2 \times 2 \times 2$ grid---\emph{(cache-resident, bandwidth-bound, device-saturated)}---was empirically demonstrated to be unreachable on plain PyTorch+MPS without kernel fusion, a finding we return to in §\ref{sec:results}.

\textbf{Selection principle.} Workloads are selected to satisfy three constraints: (1)~\emph{Resource accessibility}---executable within constrained environments ($\leq$8\,GB RAM, $\leq$10\,min total training time); (2)~\emph{Behavioral fidelity}---each workload reproduces the dominant system bottleneck of its production counterpart (e.g., attention-bound, memory-bound, pipeline-bound); (3)~\emph{Pedagogical decomposability}---each workload isolates a measurable tradeoff that students can systematically vary. Additionally, all workloads must be platform-portable (CPU, CUDA, MPS) and reproducible with fixed seeds.

Quality targets follow a documented methodology: each target is set at $0.80 \times$ the best metric achieved in 5 independent runs with default hyperparameters on reference hardware, ensuring the target is reachable within a reasonable epoch budget while leaving headroom for systems optimization. The harness auto-detects hardware at runtime (\texttt{fingerprint.py}) and stamps every submission artifact with the detected configuration, so paper claims are always traceable to measured runs.

\begin{table*}[t]
\centering
\caption{Representative MLPerf EDU workload families. $\bigstar$ = initial core families spanning distinct positions on the three-axis regime grid; see §\ref{sec:suite}. Most local reference models are inspectable PyTorch \texttt{nn.Module} implementations; the current executable registry also includes off-the-shelf SLM serving variants. Training times are hardware-stamped by the harness and should be interpreted with the report fingerprints.}
\label{tab:workloads}
\footnotesize
\renewcommand{\arraystretch}{1.25}
\setlength{\tabcolsep}{3pt}
\begin{tabular}{@{}llllrlllp{3.5cm}@{}}
\toprule
\textbf{Div.} & \textbf{Task} & \textbf{Model} & \textbf{MLPerf Map} & \textbf{Params} & \textbf{Dataset} & \textbf{Target} & \textbf{Time} & \textbf{Key Concept} \\
\midrule
Cloud & Language & NanoGPT$\;\bigstar$ & GPT-3 & 11.1M & Gutenberg TinyShakes. & Loss $<$ 2.3 & 86s & $O(N^2)$ attention, KV-cache \\
Cloud & Sparse MoE & Nano-MoE & Switch-T & 17.4M & Gutenberg TinyShakes. & Loss $<$ 0.05 & 158s & Conditional compute, gating \\
Cloud & Rec. & Micro-DLRM$\;\bigstar$ & DLRM & 23K & MovieLens & Acc $>$ 0.70 & 3s & Sparse/dense memory bound \\
Cloud & Generation & Micro-Diff. & Stable D. & 2.0M & CIFAR-10 & MSE $<$ 0.002 & 41s & Denoising steps vs.\ latency \\
Cloud & Graph & Micro-GCN & GNN & 94K & Cora & Acc $>$ 0.78 & 2s & Message passing, SpMM \\
Cloud & Text Cls. & Micro-BERT & BERT & 1.1M & SST-2 & Acc $>$ 0.75 & 45s & Bidirectional vs.\ causal attn. \\
Cloud & Time Ser. & Micro-LSTM & LSTM & 54K & ETTh1 & MSE $<$ 0.13 & 20s & Sequential bottleneck \\
Cloud & RL & Micro-RL & REINFORCE & 18K & CartPole & Rew $>$ 195 & 1s & Policy gradient variance \\
\addlinespace
Edge & Img.\ Cls. & ResNet-18$\;\bigstar$ & ResNet-50 & 11.2M & Fashion-MNIST & Top1 $>$ 0.75 & 7s & Skip connections, batch norm \\
Edge & Mobile & MobileNetV2 & MobileNet & 2.4M & Fashion-MNIST & Top1 $>$ 0.70 & 24s & Depthwise-sep.\ convolutions \\
\addlinespace
Tiny & KWS & DS-CNN$\;\bigstar$ & DS-CNN & 20K & Speech v2 & Top1 $>$ 0.90 & 51s & Spectrogram features, MCU \\
Tiny & Anomaly & Autoencoder & FC-AE & 0.3M & MNIST & MSE $<$ 0.04 & 6s & Reconstruction error \\
Tiny & Person Det. & MicroNet & VWW & 8.5K & Wake Vision & Acc $>$ 0.85 & 10s & Sub-10K model compression \\
\addlinespace
Agent & RAG & NanoRAG & --- & 20.1M & ReAct Traces & Retr.+Gen & --- & Retrieve vs.\ generate split \\
Agent & CodeGen & NanoCodeG. & --- & 13.7M & MBPP & Gen+Verify & --- & Iterative refinement loop \\
Agent & ReAct & NanoReAct$\;\bigstar$ & --- & 13.7M & ReAct Traces & Think+Act & --- & Multi-step reasoning \\
\bottomrule
\end{tabular}
\end{table*}

\begin{table}[t]
\centering
\caption{\textbf{Dataset Provenance.} Every dataset is real and citable.}
\label{tab:datasets}
\small
\renewcommand{\arraystretch}{1.25}
\begin{tabularx}{\columnwidth}{@{}l>{\raggedright\arraybackslash}Xlr@{}}
\toprule
\textbf{Dataset} & \textbf{Source} & \textbf{Loader} & \textbf{Size} \\
\midrule
TinyShakespeare & Project Gutenberg eBook 100 & Fetch + recipe & 1.1\,MB \\
Fashion-MNIST & Xiao et al., 2017 & torchvision & 83\,MB \\
CIFAR-100/10 & Krizhevsky, 2009 & Optional/proxy & 170\,MB \\
Speech Cmds v2 & Warden, 2018 & torchaudio & 2\,GB \\
MNIST & LeCun et al., 1998 & torchvision & 12\,MB \\
Wake Vision & Banbury et al., 2024 & HF/proxy & 6\,GB \\
MBPP & Austin et al., 2021 & Shipped & 50\,KB \\
\midrule
\multicolumn{4}{@{}l}{\emph{Shipped locally (real data, zero network required):}} \\
Cora & McCallum 2000 & Shipped & 168\,KB \\
SST-2 & Socher 2013 & Shipped & 3.8\,MB \\
ETTh1 & Zhou 2021 & Shipped & 850\,KB \\
MovieLens-100K & Harper 2015 & Fetch-only & 5\,MB \\
CartPole Env & Physics sim & Local & 0\,B \\
ReAct Traces & Curated & Shipped & 10\,KB \\
\bottomrule
\end{tabularx}
\end{table}

\subsection{Language, Recommendation, Graph, Time-Series, and RL Suites}

\textbf{NanoGPT} (11.1M params). A 6-layer, 6-head decoder-only transformer trained on a deterministic TinyShakespeare excerpt generated from Project Gutenberg eBook 100 (vocab=128, with the active character set induced by the corpus). The model was initially specified with a 50,257-token BPE vocabulary that inflated the parameter count to a nominal 85.9M while leaving most embedding parameters unreachable from char-level data; this mismatch was discovered and reconciled during the suite's iteration cycle, and the reported 11.1M is the honest post-reconciliation figure. The workload exposes $O(N^2)$ attention scaling and provides the compute-bound anchor of the prefill regime. Students profile attention memory growth, implement KV-cache optimization, and measure throughput-latency tradeoffs across batch sizes. A paired variant (NanoGPT-Decode, §\ref{sec:results}) surfaces the \emph{bandwidth-bound} regime on the same weights---one decode step costs as much wall-clock as the throughput of 1175 prefill tokens, the visceral demonstration of why LLM serving economics are dominated by decode. The key systems insight is that attention memory grows quadratically with sequence length, making this the primary bottleneck for serving---not model size.

\textbf{Nano-MoE} (17.4M params). A sparse Mixture-of-Experts with 8 experts and top-2 routing~\citep{shazeer2017outrageously}, where only 25\% of expert parameters activate per token. This architecture demonstrates the conditional computation pattern used in production systems such as GShard and Switch Transformer. Students compare total parameter count against active parameter count and discover that architectural choices (expert specialization) matter as much as raw scale. The model converges to loss 0.042---substantially lower than NanoGPT's 2.25 despite fewer total parameters---illustrating expert specialization. This teaches a critical systems lesson: sparsity enables better performance per FLOP, but introduces routing overhead and load-balancing complexity.

\textbf{Micro-DLRM} (23K params). A scaled-down recommendation model~\citep{naumov2019deep} with embedding tables for users (943), items (1,682), and occupations (21), a feature interaction MLP, and dense features including user demographics and 12 item genre flags from \textbf{MovieLens-100K} (100K ratings). This workload exposes the memory-bandwidth-bound vs.\ compute-bound dichotomy that defines production recommendation systems such as Meta's DLRM and Google's Wide\&Deep. \emph{Scale caveat}: at 23K parameters, the embedding tables total $943 \times 8 + 1{,}682 \times 8 + 21 \times 8 = 21{,}168$ fp32 values ($\approx$83\,KiB). At this size, the tables are intentionally small enough to avoid system-level memory-bandwidth contention; students observe the \emph{architectural} bottleneck pattern (sparse lookups vs.\ dense compute) rather than the cache-miss-driven pressure of production-scale tables. The systems takeaway is that recommendation workloads are fundamentally memory-bound---a property that persists from this micro scale to terabyte-scale production.

\textbf{Micro-Diffusion} (2.0M params). A U-Net denoising network~\citep{ho2020denoising} trained on CIFAR-10, reflecting the architecture of production diffusion systems such as Stable Diffusion and DALL-E. Inference latency scales linearly with the number of denoising steps, exposing the compute-latency tradeoff that dominates generative model serving. Students measure how step count affects generation quality and wall-clock time, learning that diffusion inference cost is a tunable parameter---a key insight for deployment.

\textbf{Micro-GCN} (94K params). A 2-layer Graph Convolutional Network~\citep{kipf2017semisupervised} for node classification on the \textbf{Cora citation network} (2,708 papers, 5,429 citations, 7 classes), reflecting the graph learning patterns used in systems such as Pinterest PinSage and Google Knowledge Graph. Implements message passing and adjacency normalization from scratch without PyG/DGL dependencies. Uses dense adjacency multiplication for pedagogical clarity; production systems use sparse representations. The model achieves 78.8\% test accuracy in 200 epochs (2s), closely matching the published Kipf \& Welling result ($\sim$81\%). The systems lesson is that irregular memory access patterns in graph workloads create fundamentally different performance characteristics from dense linear algebra.

\textbf{Micro-BERT} (1.1M params). A 2-layer bidirectional transformer for binary sentiment classification on \textbf{SST-2} (67K sentences) with word-level tokenization, reflecting the fine-tuning workflow of production NLU systems such as BERT and RoBERTa. Students compare bidirectional and causal attention patterns and analyze the pronounced train-val gap (94\% vs.\ 77\%) as a case study in overfitting and regularization. The model achieves 77.1\% validation accuracy after 20 epochs (45s on the reference platform), demonstrating that bidirectional processing enables richer representations at the cost of precluding autoregressive generation---a fundamental architectural tradeoff in systems design.

\textbf{Micro-LSTM} (54K params). A 2-layer LSTM for multivariate time-series forecasting on \textbf{ETTh1} (17,420 hourly electricity transformer readings, Zhou et al.\ 2021), predicting oil temperature from 7 sensor features with a 96$\rightarrow$24 horizon. This workload mirrors the sequential processing pattern of production forecasting systems such as Autoformer and PatchTST. Students observe classic time-series overfitting (validation MSE optimal at epoch 5, rising to 0.47 by epoch 30), motivating early stopping. The key systems insight is that sequential dependencies prevent parallelization across time steps---the fundamental bottleneck that motivates the architectural shift from RNNs to Transformers.

\textbf{Micro-RL} (18K params). A REINFORCE policy gradient agent on a pure-Python CartPole environment, reflecting the reward-based optimization pattern used in production RLHF fine-tuning pipelines. Students observe that RL reward curves are fundamentally noisier than supervised loss curves, learning to interpret stochastic optimization signals. The intentionally high variance motivates discussion of variance reduction techniques (baselines, PPO). The systems implication is that RL workloads require different convergence criteria and runtime monitoring than supervised training---a distinction critical for RLHF deployment.

\subsection{Vision and Tiny Suites}

\textbf{ResNet-18} (11.2M params). Standard ResNet~\citep{he2016} adapted for Fashion-MNIST, fully self-contained without \texttt{torchvision.models} dependency, serving as a laptop-scale proxy for the ResNet-50 workload in MLPerf Training. Students diagnose data pipeline stalls (CPU--GPU loading bottlenecks), apply augmentation and learning rate scheduling, and measure their impact on convergence rate. The current default reaches 77.3\% top-1 accuracy in 7.4 seconds on the Apple MPS reference path. The systems insight is that training throughput is often limited not by compute but by the data ingestion pipeline---a bottleneck that persists at production scale.

\textbf{MobileNetV2} (2.4M params). Inverted residual architecture with depthwise-separable convolutions~\citep{sandler2018}, fully self-contained, reflecting the mobile inference workloads in MLPerf Inference (Edge). The depthwise/pointwise factorization reduces parameters by $5\times$ vs.\ ResNet-18 with minimal accuracy loss. Students compare parameter efficiency and inference throughput, learning that architectural factorization can dramatically reduce deployment cost---the core insight behind efficient edge inference.

\textbf{DS-CNN} (20K params). A depthwise-separable CNN~\citep{zhang2017hello} for 12-class keyword spotting on Speech Commands v2~\citep{warden2018speech}, processing mel spectrograms (40$\times$101 frames), reflecting the DS-CNN benchmark in MLPerf Tiny. Students quantize the model to INT8 and measure compression ratios, learning that the depthwise-separable factorization ($C \!\times\! K^2 + C \!\times\! C'$ vs.\ standard $C \!\times\! C' \!\times\! K^2$) is the key to MCU-class deployment under extreme memory constraints.

\textbf{Anomaly Detection AE} (0.3M params). An 8-dim bottleneck autoencoder trained on MNIST digit ``0''; anomalous digits (1--9) produce elevated reconstruction error, mirroring the MLPerf Tiny anomaly detection benchmark used in industrial settings such as factory machine sound monitoring~\citep{banbury2021mlperf}. Students tune the anomaly detection threshold to maximize AUC and explore precision-recall tradeoffs. The systems insight is that unsupervised anomaly detection shifts the evaluation challenge from accuracy to threshold calibration---a fundamentally different systems problem from supervised classification.

\textbf{Wake Vision VWW} (8.5K params). A MicroNet CNN with depthwise-separable convolutions for binary person detection, trained on a pedagogical subset of Wake Vision~\citep{banbury2024wakevision} (5K images) with a CIFAR-10 proxy fallback, reflecting the MLPerf Tiny Visual Wake Words benchmark. Students explore the impact of aggressive model compression on accuracy and measure inference latency at microcontroller scale. The model achieves 87.3\% validation accuracy in approximately 10 seconds, demonstrating that meaningful person detection is achievable with fewer than 10K parameters---a systems lesson in the lower bounds of model compression.

\subsection{Agent Suite}
\label{sec:agents}

Existing agent benchmarks (SWE-bench, AgentBench) measure \emph{capability}: can the agent solve the task? MLPerf EDU's agent workloads measure \emph{systems performance}: latency, memory growth, and throughput of agentic pipelines. This distinction is critical for deployment, where the cost-per-query of an agentic system can exceed $100\times$ that of a single model call.

\textbf{NanoRAG} (20.1M params)~\citep{lewis2020retrieval}. Brute-force TF-IDF retrieval over a document corpus followed by generation, reflecting the retrieve-then-generate architecture used in enterprise search-augmented systems such as Perplexity and Bing Chat. Students measure the retrieve-vs.-generate latency split, learning that retrieval overhead can dominate total query time---a critical insight for RAG system design.

\textbf{NanoCodeGen} (13.7M params)~\citep{chen2021evaluating}. Iterative code generation with unit-test verification, trained on MBPP~\citep{austin2021mbpp} (20 real Python programming tasks), reflecting the generate-then-verify loop used in systems such as GitHub Copilot and Cursor. The 20-task subset is sufficient for measuring systems throughput (tokens-per-attempt, verification overhead) but insufficient for statistically significant code generation accuracy; students should interpret pass@1 as a qualitative signal. The systems lesson is that iterative refinement multiplies inference cost linearly with attempt count.

\textbf{NanoReAct} (13.7M params)~\citep{yao2023react}. Multi-step reasoning with a dual-head transformer (LM head + tool-selection head), trained on 25 structured reasoning traces, reflecting the tool-augmented agent pattern used in systems such as ChatGPT plugins and function-calling APIs. Students observe expected quadratic context growth as each Think$\rightarrow$Act$\rightarrow$Observe cycle appends tokens proportional to the reasoning trace, with per-step memory tracking. \emph{Important caveat}: the model learns to reproduce reasoning trace \emph{patterns}, not to perform genuine logical reasoning; in production, ReAct relies on pre-trained LLM knowledge. This distinction between imitation and reasoning is itself pedagogically valuable, teaching students to separate systems evaluation from capability evaluation.

All agent quality targets were verified in 5 independent runs on reference hardware. NanoRAG achieves 68\% retrieval accuracy (target: $>$60\%), NanoCodeGen achieves 20\% pass@1 (target: $>$15\%), and NanoReAct achieves 78\% trace reproduction accuracy (target: $>$70\%). Agent workloads are trained on structured traces using teacher-forcing, producing standard cross-entropy loss curves; however, their primary evaluation is inference-time systems metrics (latency, memory, throughput).

% ============================================================================
\section{Experimental Results}
\label{sec:results}
% ============================================================================

We report an initial validation snapshot for 13 supervised workload families on a consumer Apple Silicon laptop using the MPS backend, without a dedicated GPU. Agent and serving workloads are verified through inference-time metrics. Hardware is auto-detected by the harness and stamped into every artifact. \Cref{fig:training-curves} shows convergence curves; \Cref{tab:results} reports final metrics for this measured snapshot.

\begin{figure*}[t]
\centering
\includegraphics[width=0.95\textwidth]{figures/training_curves.pdf}
\caption{\textbf{Training Convergence Curves.} Train (solid) and validation (dashed) loss for representative workloads over 20--30 epochs on Apple Silicon (MPS backend). All models converge on real datasets. (a) NanoGPT: steady character-level convergence on the Project Gutenberg TinyShakespeare recipe. (b) Nano-MoE: rapid expert specialization. (c) ResNet-18: gradual Fashion-MNIST feature learning. (d) Micro-DLRM: recommendation on MovieLens-100K reaching 70.5\% accuracy. (e) Micro-Diffusion: U-Net reconstruction reaching MSE $<$ 0.002. (f) Anomaly AE: val loss intentionally higher for anomaly detection. Additional workloads (GCN on Cora, BERT on SST-2, LSTM on ETTh1) show characteristic overfitting patterns. Total supervised suite training time: 552 seconds.}
\label{fig:training-curves}
\end{figure*}


\begin{table}[t]
\centering
\caption{\textbf{Final Training Metrics.} All supervised workloads converge on Apple Silicon (MPS backend). Gap $=$ Val $-$ Train loss. $\dagger$AE: val loss is intentionally higher (anomalous inputs). $\ddagger$Accuracy where applicable.}
\label{tab:results}
\small
\renewcommand{\arraystretch}{1.25}
\begin{tabularx}{\columnwidth}{@{}lrrrrl@{}}
\toprule
\textbf{Workload} & \textbf{Train} & \textbf{Val} & \textbf{Gap} & \textbf{Acc.$^\ddagger$} & \textbf{Time} \\
\midrule
NanoGPT & 2.248 & 2.205 & $-$0.043 & --- & 89s \\
Nano-MoE & 0.042 & 0.042 & +0.000 & --- & 158s \\
ResNet-18 & 2.456 & 2.485 & +0.029 & 36.3\% & 64s \\
DLRM & 0.535 & 0.573 & +0.038 & 70.5\% & 3s \\
Diffusion & 0.002 & 0.000 & $-$0.001 & --- & 41s \\
AE$^\dagger$ & 0.030 & 0.064 & +0.034 & --- & 6s \\
DS-CNN & 0.916 & 1.026 & +0.110 & 71.2\% & 51s \\
VWW & 0.310 & 0.330 & +0.020 & 87.3\% & 10s \\
GCN & 0.052 & 0.981 & +0.929 & 81.6\% & 2s \\
BERT & 0.130 & 0.874 & +0.744 & 77.1\% & 45s \\
LSTM & 0.019 & 0.125 & +0.106 & --- & 20s \\
RL & 0.087 & --- & --- & 195+ & 1s \\
MobileNetV2 & 1.983 & 2.176 & +0.193 & 41.2\% & 60s \\
\midrule
\textbf{Total} & & & & & \textbf{552s} \\
\bottomrule
\end{tabularx}
\end{table}

\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{figures/convergence_summary.pdf}
\caption{\textbf{Convergence Summary (13 Supervised Workload Families).} (a--d) Representative training curves showing four distinct convergence patterns: NanoGPT exhibits strong overfitting (shaded red), LSTM shows classic time-series overfitting with early val plateau, Diffusion converges tightly, and VWW demonstrates noisy small-dataset behavior. (e) Final train vs.\ val loss for the measured supervised snapshot; vertical dashed lines separate the initial language/recommendation, vision, and tiny groupings. (f) Training time on Apple Silicon for the measured snapshot. Agent and serving workloads measure inference-time systems metrics and are summarized separately in \S\ref{sec:arch_impl}.}
\label{fig:summary}
\end{figure*}

\textbf{Analysis of convergence behavior.} Several observations merit discussion. First, Nano-MoE converges to loss 0.042, substantially lower than NanoGPT's 2.25, despite an active-parameter count (top-2 routing across 8 experts, so $\sim$25\% of the 17.4M total = $\sim$4.4M active per token) well below NanoGPT's 11.1M. This demonstrates expert specialization: conditional compute delivers better loss per active FLOP than dense compute at the same dataset. Second, DLRM achieves 70.5\% validation accuracy on MovieLens-100K with a realistic train-val gap (0.038), confirming genuine collaborative filtering signal even at micro scale. Third, Micro-GCN achieves 81.6\% test accuracy on Cora, closely matching the published Kipf \& Welling result ($\sim$81\%), validating the from-scratch implementation against established baselines. Fourth, Micro-BERT on SST-2 exhibits a pronounced train-val gap (94\% vs.\ 77\%), creating an instructive overfitting case study. Fifth, Micro-LSTM on ETTh1 displays classic time-series overfitting (validation MSE optimal at epoch 5, then rising), motivating early stopping. Sixth, the complete 13-workload supervised suite finishes in under 10 minutes on consumer hardware, satisfying the resource accessibility constraint.

\begin{figure}[t]
\centering
\includegraphics[width=\columnwidth]{figures/model_scale.pdf}
\caption{\textbf{Model Scale Reduction.} MLPerf EDU reduces model parameters by $100\text{--}5000\times$ vs.\ official MLPerf while preserving architectural patterns. Log scale.}
\label{fig:scale}
\end{figure}

\subsection{Inference Latency}

We validate the inference harness by measuring SingleStream latency for four representative workloads (\Cref{tab:inference}). All measurements use batch size 1 on Apple Silicon CPU (no GPU) over 200 queries per workload (100 for NanoGPT). The results reveal the expected performance hierarchy: DLRM's sparse lookups dominate at 33K QPS, DS-CNN's lightweight convolutions achieve 2K QPS, and NanoGPT's attention mechanism is the bottleneck at 163 QPS. These differences give students a concrete lesson in where compute time is actually spent.

\begin{table}[t]
\centering
\caption{Inference latency (SingleStream, batch=1, Apple Silicon CPU). Values: median $\pm$ std over 5 runs of 100 queries each. CV $<$ 5\% for all workloads.}
\label{tab:inference}
\footnotesize
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{@{}lrrrr@{}}
\toprule
\textbf{Workload} & \textbf{p50 (ms)} & \textbf{p99 (ms)} & \textbf{CV} & \textbf{QPS} \\
\midrule
NanoGPT-6L & 5.90{\tiny$\pm$0.04} & 7.49{\tiny$\pm$1.36} & 0.6\% & 163 \\
ResNet-18  & 3.54{\tiny$\pm$0.02} & 3.75{\tiny$\pm$0.12} & 0.6\% & 279 \\
DS-CNN     & 0.47{\tiny$\pm$0.02} & 0.55{\tiny$\pm$0.05} & 4.0\% & 2,024 \\
Micro-DLRM & 0.03{\tiny$\pm$0.00} & 0.03{\tiny$\pm$0.00} & 0.7\% & 33,219 \\
\bottomrule
\end{tabular}
\end{table}

\textbf{Bottleneck analysis.} The $200\times$ latency gap between DLRM (0.03\,ms) and NanoGPT (5.90\,ms) reflects fundamentally different positions on the three-axis regime grid. DLRM's forward pass is dominated by embedding lookups (3 tables of $\sim$1K entries $\times$ 32-dim)---pure memory reads with minimal arithmetic, yielding an arithmetic intensity $<$ 1 FLOP/byte. NanoGPT's prefill self-attention computes $QK^T$ matrices ($O(N^2 d)$ FLOPs for sequence length $N$, head dim $d$), making it compute-bound with arithmetic intensity $\gg$ 10 FLOP/byte. This difference is directly observable in student profiling exercises.

\textbf{An unreachable cell, made explicit.} A naive reading of the three-axis grid would predict every combination of (cache-resident, DRAM-bound) $\times$ (compute-bound, bandwidth-bound) $\times$ (device-saturated, dispatch-bound) as a reachable workload configuration. Our experimental sweep found otherwise: the cell \emph{(cache-resident, bandwidth-bound, device-saturated)} is structurally unreachable on plain PyTorch with the MPS backend at any batch size we tested. The reason is mechanical: a workload cannot simultaneously fit its per-step working set in LLC \emph{and} saturate DRAM bandwidth \emph{and} amortize PyTorch's 50--200\,µs per-kernel dispatch overhead, because the first two requirements force a working-set size in the tens-of-megabytes band where our reference LLC is exceeded. Production LLM serving stacks (vLLM, TensorRT-LLM, llama.cpp, MLX) close this gap with kernel fusion and custom Metal/CUDA kernels; the gap between those stacks and stock PyTorch is itself the iteration's pedagogical lesson.

\textbf{Measurement-driven reclassification.} An earlier draft of this paper assigned regime labels by static analysis (counting weights, estimating activation peaks). Running every workload through a roofline-emitter harness (\S\ref{sec:design}) that wraps each forward pass and records (achieved FLOPS, achieved bandwidth, dispatch utilization) revealed that 15 of our initial labels did not survive measurement and were revised. Notable surprises: ResNet-18 at \texttt{fp32}, \texttt{B=8} on M5 Max is bandwidth-bound (intensity 3.9 FLOP/byte), not compute-bound as the textbook framing would suggest; Micro-DLRM at small batch is compute-bound on the MLP despite the architectural sparse-gather pattern being canonically memory-bound; and Micro-Diffusion is bandwidth-bound, matching MobileNet-class arithmetic-intensity intuition rather than the "generative model" mental model. The three-axis taxonomy's insistence on an \texttt{unmeasured} value for any axis without measurement evidence prevented these mislabels from propagating into the published result tables.

% ============================================================================
\section{Educational Integration}
\label{sec:pedagogy}
% ============================================================================

We present three concrete classroom exercises demonstrating how MLPerf EDU enables systems ML education. These scenarios were developed through iterative design with ML systems instructors. \Cref{tab:competencies} maps each canonical core workload to specific, measurable learning competencies classified by Bloom's revised taxonomy~\citep{anderson2001taxonomy}.

\begin{table}[t]
\centering
\caption{\textbf{Competency Matrix.} Core workloads mapped to learning outcomes. Bloom's levels: \textbf{Ap}ply (L3), \textbf{An}alyze (L4), \textbf{Ev}aluate (L5), \textbf{Cr}eate (L6).}
\label{tab:competencies}
\footnotesize
\renewcommand{\arraystretch}{1.2}
\begin{tabularx}{\columnwidth}{@{}lXc@{}}
\toprule
\textbf{Workload} & \textbf{Competency} & \textbf{Level} \\
\midrule
NanoGPT & Profile attention memory scaling ($O(N^2)$ vs.\ $N$) & An \\
 & Implement KV-cache and measure speedup & Ap \\
 & Evaluate batch size vs.\ throughput tradeoff & Ev \\
\addlinespace
Micro-DLRM & Identify memory-bandwidth bottleneck via profiling & An \\
 & Compare sparse lookup vs.\ dense compute cost & Ev \\
\addlinespace
ResNet-18 & Diagnose data pipeline stalls (CPU vs.\ GPU) & An \\
 & Apply augmentation, LR schedule, AMP & Ap \\
\addlinespace
DS-CNN & Measure depthwise vs.\ standard conv efficiency & An \\
 & Quantize model to INT8 and measure compression & Ap \\
\addlinespace
NanoReAct & Measure per-step memory growth (context window) & An \\
 & Evaluate latency scaling with reasoning depth & Ev \\
\addlinespace
\emph{Capstone} & Design a novel workload using the evaluation tuple & Cr \\
\bottomrule
\end{tabularx}
\end{table}

\subsection{Example 1: Systems Optimization Challenge}

\textbf{Setting.} Students receive a ``broken baseline'' ResNet-18 with intentionally poor configuration: \texttt{num\_workers=0} (no data parallelism), no learning rate schedule, no data augmentation, batch size 8. Their task: apply systems optimizations to maximize accuracy within a fixed wall-clock budget.

\begin{lstlisting}[caption={\textbf{Example 1: Optimization.} Students diagnose and fix systems bottlenecks.},label={lst:lab1}]
# Student starts with broken baseline:
train_ld, val_ld = get_dataloaders(
    "resnet18", batch_size=8)  # Too small!
opt = torch.optim.SGD(model.parameters(),
    lr=0.1)  # No schedule!

# Task: Profile, identify bottleneck, fix
# Expected sequence:
# 1. Increase batch_size -> 64 (4x throughput)
# 2. Add CosineAnnealingLR (convergence)
# 3. Add RandomCrop + HorizontalFlip (generalization)
# 4. Enable torch.amp (2x throughput on GPU)
# Final: accuracy 5% -> 36% -> 50%+
\end{lstlisting}

\textbf{Learning outcomes.} Students discover that: (i) DataLoader parallelism (\texttt{num\_workers}) eliminates CPU-GPU pipeline stalls, (ii) batch size affects both throughput and convergence stability, (iii) augmentation acts as regularization, reducing the train-val gap.

\subsection{Example 2: Cross-Architecture Comparison}

\textbf{Setting.} Students train NanoGPT (dense) and Nano-MoE (sparse) on identical data and compare convergence speed, memory usage, and per-token latency.

\begin{lstlisting}[caption={\textbf{Example 2: Dense vs.\ Sparse.} Same data, different architectures.},label={lst:lab2}]
# Train both on TinyShakespeare
for model_name in ["nanogpt-12m", "nano-moe-12m"]:
    t_ld, v_ld = get_dataloaders(model_name)
    model = load_model(model_name)
    # ... train for 25 epochs ...
    # Measure:
    #   NanoGPT: 11.1M params (char-level), loss 2.25
    #   Nano-MoE: 17.4M params, loss 0.04
    # Discussion: Why does MoE converge 50x
    # lower with fewer parameters?
    # Answer: Expert routing enables
    # specialization + overparameterized
    # individual experts.
\end{lstlisting}

\textbf{Learning outcomes.} Students discover that total parameter count alone does not explain convergence quality; architectural choices (sparse routing, expert specialization) matter as much as raw scale.

\subsection{Example 3: Anomaly Detection Threshold Tuning}

\textbf{Setting.} Students train the autoencoder on ``normal'' MNIST digits, then tune the anomaly detection threshold to maximize AUC on the validation set.

\begin{lstlisting}[caption={\textbf{Example 3: Anomaly Detection.} Reconstruction error as a signal.},label={lst:lab3}]
model = AnomalyDetectionAE(input_dim=784)
train_ld, val_ld = get_dataloaders("anomaly-ae")
# ... train on digit 0 only ...

# Inference: compute anomaly scores
scores = model.anomaly_score(val_batch)
# Normal (digit 0): score ~ 0.03
# Anomalous (digit 7): score ~ 0.12

# Student task: find threshold T such that
# AUC is maximized. Explore precision-recall
# tradeoff. Connect to industrial use:
# factory machine sound anomaly detection.
\end{lstlisting}

The canonical core provides sufficient structure for a 5-week accelerated module; the full suite with inference exercises supports a 15-week graduate course. Detailed weekly schedules and deployment guidance are provided in the repository documentation.

% ============================================================================
\section{Architecture and Implementation}
\label{sec:architecture}
% ============================================================================

\subsection{Unified Dataset Factory}

All data loading is centralized through \texttt{dataset\_factory.py} (\Cref{lst:factory}). The factory accepts a model name and returns deterministic \texttt{(train\_loader, val\_loader)} with seed=42 and 80/20 splits. This guarantees identical data ordering for any student, enabling reproducible gradient trajectories.

\subsection{Model Registry}

All models are registered in a central dictionary mapping workload names to \texttt{(module, class)} tuples, enabling dynamic instantiation:

\begin{lstlisting}[caption={\textbf{Model Registry.} Dynamic model loading.}]
MODEL_REGISTRY = {
  "nanogpt-12m": ("reference.cloud.nanogpt_train",
                   "NanoGPTWhiteBox"),
  "dscnn-kws":   ("reference.tiny.dscnn_kws",
                   "DSCNN"),
  "anomaly-ae":  ("reference.tiny.anomaly_detection_ae",
                   "AnomalyDetectionAE"),
  # ... 16 total workloads
}
\end{lstlisting}

\subsection{Load Generator and Inference Harness}

For inference benchmarking, MLPerf EDU includes a Python-native \texttt{LoadGenProxy} that implements all four MLPerf Inference scenarios~\citep{reddi2020mlperf}. Each scenario models a distinct deployment pattern: \textbf{Offline} models bulk throughput by issuing the entire query set at once and measuring total processing time. \textbf{SingleStream} models serial latency by issuing one query at a time and measuring per-query response time. \textbf{Server} models queuing under stochastic arrivals by generating queries with exponentially distributed inter-arrival times at a configurable QPS, where the SUT must meet a p99 latency bound. \textbf{MultiStream} models coordinated concurrent queries, as in multi-camera edge inference. Unlike the official C++ LoadGen~\citep{reddi2020mlperf}, our implementation uses \texttt{asyncio} for scenario scheduling. This sacrifices sub-microsecond timing precision but makes the load-generation logic fully inspectable---students can step through the scheduling loop and observe queuing behavior directly.

Samples flow through the system as follows. The load generator creates \texttt{QuerySample} objects with unique IDs and precise arrival timestamps. Each sample is dispatched to the student's System Under Test (SUT) plugin, and the response latency is recorded. After the run completes, the harness computes p50/p90/p95/p99 latency percentiles, throughput (QPS), and energy (Joules).

\subsection{Power Measurement}

The harness can add aggregate energy metadata to reports when \texttt{--power} is requested. The default path records an explicit nominal estimate so course runs do not require privileged hardware counters. Future integrations can replace that source with platform counters such as macOS \texttt{powermetrics}, Linux GPU telemetry, IOKit energy counters, or external meters while preserving the report schema.

\emph{Limitations}: The default power path is an estimate, not a calibrated hardware measurement. It is useful for teaching report structure and for controlled relative experiments, but absolute Joule claims should use documented hardware-counter or external-meter sources. Optional counter integrations may require elevated privileges or platform-specific setup and should be recorded explicitly in the run manifest.

\subsection{Submission Artifacts}

After each run, the current harness writes a workload or aggregate report plus sidecars in the selected output directory. The default artifact set includes:
\begin{itemize}[leftmargin=*, itemsep=1pt]
\item A JSON report containing workload, suite, profile, scenario, backend, metrics, quality metadata, and public status.
\item A human-readable HTML report for course inspection and artifact review.
\item A CSV report for spreadsheets and lightweight analysis.
\item A \texttt{.provd.json} provenance manifest verified by \texttt{mlperf verify}.
\item A run fingerprint covering hardware, software, execution profile, and data modes.
\item Dataset/model asset dossiers, public release status, and source/license metadata.
\item Checkpoint lineage for checkpoint-backed inference workloads, including source quality and artifact policy.
\end{itemize}

This mirrors the MLPerf submission process at a course-friendly scale: students are ``submitters'' whose results can be verified, compared, packaged, and graded reproducibly. Rather than expose Closed/Open division language as the primary interface, the executable contract uses three profiles. \texttt{min} is a quick setup/correctness path, \texttt{max} is the standard comparable benchmark path, and \texttt{pro} is the research envelope for repetitions, optimization variants, backend sweeps, and ablations.

\textbf{Auditability and anti-reward-hacking design.} Each benchmark run produces a verifiable execution trace consisting of dataset hashes, model checkpoint hashes, timestamped training logs, and latency measurements. These artifacts enable third-party auditing of results and reduce the risk of fabricated submissions. Specifically, the system prevents three classes of gaming: (1)~\emph{result fabrication}---SHA-256 hashes tie reported metrics to actual computation, making it infeasible to report results without executing the workload; (2)~\emph{cross-machine fraud}---hardware fingerprints embedded in each submission detect results transferred from more powerful hardware; (3)~\emph{shortcut computation}---the compliance checker verifies that the reported dataset hash matches the canonical dataset, preventing students from training on simplified data.

\textbf{Run policy.} Score-bearing workloads with \texttt{target\_basis: reference\_runs} must declare a reference protocol: profile, backend, machine class, dataset mode, seeds, aggregation statistic, artifact policy, and rerun policy. The current public-result slice uses five reference runs and reports the median. Reports surface this protocol directly in JSON, CSV, and HTML.

\textbf{Public audit.} \texttt{mlperf audit} separates development blockers from public-release warnings. Development audit checks registry completeness, public status, scenarios, runner coverage, asset dossiers, quality targets, functional checks, and checkpoint dependencies. \texttt{mlperf audit --policy public} additionally treats unresolved dataset/model release warnings as release blockers until maintainers or MLCommons reviewers accept the policy.

\subsection{CLI as Evaluation Tuple Instantiation}

The benchmark exposes a unified command-line interface that serves as the concrete instantiation of the evaluation tuple $\mathcal{S} \rightarrow \mathcal{M}$. The public executable is \texttt{mlperf}; the suite defaults to \texttt{mlperf-edu}. Commands use only suite, profile, workload, and variant as selection concepts. A canonical workload selects all variants; adding \texttt{--variant} selects exactly one row.

\begin{lstlisting}[caption={CLI usage: each command instantiates the evaluation tuple.},label={lst:cli}]
# Setup and inspect
$ mlperf init --profile min
$ mlperf list --profile max
$ mlperf fetch --workload nanogpt-train --profile max
$ mlperf audit --policy public

# Run one workload or a canonical workload family
$ mlperf run --workload nanogpt-train --profile min
$ mlperf run --workload smollm2-chat-inference \
    --variant quantized-int8 --profile max

# Review and verify artifacts
$ mlperf report submissions/latest.json --format html
$ mlperf verify submissions/latest.provd.json
$ mlperf grade submissions/
\end{lstlisting}

The registry remains declarative: the native \texttt{registry/suites/...} layout records suites, canonical workload names, variants, model/data sources, quality targets, reference protocols, runners, public status, and provenance; \texttt{workloads.yaml} is generated as a compatibility mirror. Each field maps directly to a component of the evaluation tuple:

\begin{lstlisting}[caption={\textbf{YAML Config.} Each field maps to $\mathcal{S}$.}]
workload: nanogpt-train
model: nanogpt-12m             # S.Model
dataset: tinyshakespeare       # S.Data
scenario: single_stream        # S.Scenario
quality_target:
  metric: cross_entropy_loss   # S.Constraints
  value: 2.3
  target_basis: reference_runs
public:
  status: score-bearing
runner:
  min: mlperf.runners.nanogpt:run_min
  max: mlperf.runners.nanogpt:run_max
\end{lstlisting}

This design provides two complementary interfaces: an imperative CLI for rapid iteration and a declarative registry for reproducible experiment management. Both map explicitly to $\mathcal{S} \rightarrow \mathcal{M}$, reinforcing the evaluation tuple abstraction at every interaction point. Variants such as dynamic INT8 SLM serving, batching, long-context serving, speculative decoding, and dtype changes expose optimization studies without creating a separate top-level workload for every experiment.

% ============================================================================
\section{Validation Methodology}
\label{sec:feedback}
% ============================================================================

Our current validation method combines executable checks with structured review packets. The executable side runs \texttt{mlperf audit}, \texttt{mlperf validate smoke}, \texttt{mlperf validate coverage}, \texttt{mlperf validate max}, unit tests, and provenance verification. The review side generates one-page packets for public-result candidates so instructors, architecture researchers, artifact evaluators, and MLCommons reviewers can inspect commands, assets, quality targets, warnings, and provenance without reading the whole repository. A formal classroom and external-review study remains future work; the criteria below are the review dimensions we use before promoting a workload.

\textbf{Architectural fidelity.} A workload should preserve the systems pattern it claims to teach. NanoGPT exposes quadratic attention and checkpoint-backed serving phases, Micro-DLRM exposes sparse lookup and dense interaction behavior, and DS-CNN exposes depthwise-separable convolution. The models are intentionally small, but the bottleneck story must be honest.

\textbf{Hardware-agnostic grading.} Public score-bearing workloads use task-quality targets rather than raw runtime thresholds. This keeps correctness comparable across CPU, MPS, and CUDA systems while still recording platform-specific latency, throughput, and energy metadata.

\textbf{Inference harness completeness.} The harness separates workload selection, asset fetching, scenario execution, quality or functional checks, report generation, and provenance verification. This mirrors the useful structure of MLPerf while remaining inspectable enough for students.

\textbf{Data provenance credibility.} Public-result candidates must expose dataset and model source, citation, license status, public-release status, cache behavior, and next-step policy. Synthetic, tiny, or micro-sharded data can be useful for \texttt{min} and systems-only runs, but reports must label it clearly and must not present it as a public score.

% ============================================================================
\section{Related Work}
\label{sec:related}
% ============================================================================

\textbf{MLPerf.} The official MLPerf Training, Inference, and Tiny suites~\citep{mattson2020, reddi2020mlperf, banbury2021mlperf} set the industry standard. MLPerf EDU inherits their architectural diversity and measurement discipline while reducing the hardware requirement from enterprise clusters to a single laptop. \Cref{tab:comparison} summarizes the positioning.

\textbf{DAWNBench.} Introduced the time-to-accuracy and cost-to-accuracy metrics~\citep{coleman2019} that influenced MLPerf's quality-target design. DAWNBench focused on cloud cost optimization; MLPerf EDU focuses on pedagogical accessibility.

\textbf{TorchBench.} Covers 80+ PyTorch models for regression testing within the framework development pipeline~\citep{ansel2024torchbench}. Unlike MLPerf EDU, it is not designed for education and does not include datasets, quality targets, or a load generator.

\textbf{AIBench and BenchCouncil.} The AIBench suite~\citep{gao2021aibench} provides comprehensive AI workloads across domains including datacenter, edge, and IoT. While AIBench targets industrial deployment characterization, MLPerf EDU targets pedagogical understanding with white-box implementations.

\textbf{Educational ML frameworks.} TinyTorch teaches framework internals through bottom-up construction. Fast.ai and d2l.ai focus on algorithmic understanding. MLPerf EDU is complementary: it provides workloads that such frameworks benchmark, and its focus is on systems performance measurement rather than ML algorithms.

\textbf{Agent benchmarks.} SWE-bench~\citep{jimenez2024swebench} and AgentBench measure agent \emph{capability} (task completion rate). MLPerf EDU's agent workloads are orthogonal: they measure the \emph{systems efficiency} (latency, memory, power) of agentic pipelines, a distinction critical for production deployment.

\textbf{Non-ML benchmarks.} The SPEC CPU~\citep{henning2006} and TPC~\citep{poess2000} benchmark suites provide relevant methodological precedent for educational benchmarking. SPEC's emphasis on reproducible measurement conditions and TPC's multi-dimensional metrics (throughput, price-performance, energy) informed MLPerf EDU's evaluation tuple design. Our work transfers these established benchmarking principles to the ML systems domain.

\begin{table}[t]
\centering
\caption{Comparison with existing benchmark suites.}
\label{tab:comparison}
\footnotesize
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{@{}lcccccc@{}}
\toprule
 & \rotatebox{70}{MLPerf} & \rotatebox{70}{DAWNBench} & \rotatebox{70}{TorchBench} & \rotatebox{70}{MLPerf Tiny} & \rotatebox{70}{\textbf{Ours}} \\
\midrule
Single laptop       &   & \checkmark &   & \checkmark & \checkmark \\
White-box models    &   &   &   & \checkmark & \checkmark \\
Local datasets      &   &   &   &   & \checkmark \\
Load generator      & \checkmark &   &   &   & \checkmark \\
Power measurement   & \checkmark &   &   &   & (\checkmark) \\
Agent workloads     &   &   &   &   & \checkmark \\
Compliance checker  & \checkmark &   &   & \checkmark & \checkmark \\
Submission format   & \checkmark &   &   & \checkmark & \checkmark \\
Modalities covered  & 6 & 2 & 10+ & 3 & \textbf{10} \\
\bottomrule
\end{tabular}
\end{table}

% ============================================================================
\section{Limitations and Future Work}
\label{sec:limitations}
% ============================================================================

The GCN, BERT, and LSTM workloads all use real academic datasets (Cora, SST-2, ETTh1), which means models show realistic overfitting patterns rather than trivial convergence. While these datasets are well-established benchmarks in their respective fields, they are smaller than production datasets, so the absolute accuracy numbers (78.8\%, 77.1\%, MSE 0.125) are generally lower than state-of-the-art results on these benchmarks obtained with larger models.

The REINFORCE agent for the RL workload is intentionally high-variance to demonstrate RL's difficulty, but students may struggle to reach the 195-reward quality target. Adding a PPO baseline would provide a more stable alternative.

All datasets are real, citable data with provenance. However, some datasets (e.g., MovieLens-100K for DLRM) are small enough that the embedding tables are correspondingly small ($\approx$83\,KiB in float32 for the user, item, and occupation tables vs.\ terabytes in production), which means the memory-bandwidth contention characteristic of production recommendation systems is not reproduced. The \emph{architectural} pattern---sparse lookups vs.\ dense compute---remains observable.

Power and energy reporting currently prioritizes unprivileged aggregate estimates. More accurate platform counters or external meters are future work and should be added without changing the report schema.

NanoGPT and Nano-MoE both train on TinyShakespeare (1.1\,MB). While this enables a clean controlled comparison (Example 2), it means neither workload experiences data-scale bottlenecks. The framework prioritizes \emph{compute pattern} realism (attention scaling, expert routing) over data-scale realism---a deliberate design choice for laptop-constrained environments.

NanoGPT uses character-level tokenization (vocab size 65) rather than BPE (50,257 tokens in production GPT-2). This simplification means students do not encounter subword tokenization effects, but it preserves the $O(N^2)$ attention bottleneck that is the primary systems concern. An optional BPE tokenization exercise is planned as a future extension.

\subsection{Threats to Validity}

\textbf{Internal validity.} Models with inherently stochastic training (Micro-RL, NanoReAct) may show significant run-to-run variance. We mitigate this by reporting median metrics across 5 independent runs with fixed seed, following MLPerf convention. Unlike production MLPerf, which requires convergence within a fixed epoch budget, MLPerf EDU relaxes this constraint; this is a deliberate design choice because the pedagogical objective is understanding convergence behavior, not establishing competitive benchmarks.

\textbf{External validity.} Micro-scale models are intentionally reduced until the dominant architectural pattern remains observable under accessible hardware constraints. For example, DLRM's embedding tables ($\approx$83\,KiB) do not reproduce terabyte-scale memory-bandwidth contention, but the sparse-vs.-dense compute pattern is preserved. This is a deliberate teaching choice, not a limitation: some workloads preserve compute patterns (NanoGPT's $O(N^2)$ attention, Nano-MoE's conditional routing) while others preserve architectural structure without system-level pressure. All results are tied to auto-detected hardware fingerprints. Convergence behavior (loss curves) is hardware-independent; absolute timings and power measurements are platform-specific. Cross-hardware validation (NVIDIA CUDA, Intel CPU) is planned; hardware fingerprints in submission artifacts enable future cross-platform comparison.

\textbf{Construct validity.} The framework assumes that understanding microscale bottlenecks transfers to production-scale reasoning. That assumption is plausible for compute-pattern and memory-access-pattern reasoning, but it still needs external validation through course pilots, artifact-evaluation use, and MLCommons review. Scale-dependent effects such as communication overhead and multi-device coordination are explicitly documented as out of scope for the first laptop-class release. A formal classroom evaluation with pre/post competency assessment is planned for a future course offering.

% ============================================================================
\section{Conclusion}
\label{sec:conclusion}
% ============================================================================

MLPerf EDU demonstrates that the architectural diversity and measurement rigor of industry-standard ML benchmarks can be made accessible to students without sacrificing methodological integrity. The current executable suite contains 30 workload rows across 10 product-oriented suites, with five score-bearing workloads, four performance-bearing workloads, and a broader set of systems-only teaching and research variants. The implementation is intentionally pragmatic: some workloads are pure local PyTorch references, while the SLM suite uses permissive off-the-shelf Hugging Face models to expose modern serving behaviors such as quantization, batching, and long-context decode.

The primary intellectual contribution is not the benchmarks themselves but the \emph{evaluation tuple} abstraction $\mathcal{S} \rightarrow \mathcal{M}$ that unifies them. By teaching students to reason about (Model, Data, Hardware, Scenario, Constraints) $\rightarrow$ (Latency, Throughput, Accuracy, Energy), MLPerf EDU provides a transferable framework for systems evaluation that outlasts any specific model or dataset. We believe this ``benchmarking literacy'' is as fundamental to AI systems education as algorithm design is to computer science.

\subsection*{Reproducibility}
All code, workload metadata, reference runners, generated review packets, and documentation are available at \url{https://github.com/harvard-edge/mlperf-edu}. The executable entry point is \texttt{mlperf}. Runs write JSON, HTML, CSV, and \texttt{.provd.json} artifacts by default. Hardware and software fingerprints are stamped into reports and provenance manifests. Dataset and model dossiers record source URL, citation, license status, public-release status, and expected cache behavior. Public-result review currently depends on resolving or explicitly accepting the remaining MovieLens-100K release warning; Project Gutenberg TinyShakespeare, Fashion-MNIST, MNIST, and bundled prompt fixtures have explicit release policies in the current structured audit.

\bibliography{refs}

\end{document}