Note
📌 Early release (2026)
MLSys·im shipped with the 2026 MLSysBook refresh. The analytical modeling framework, APIs, and lab integrations are actively iterated as we harden the package and teaching workflows.
Feedback — GitHub issues or pull requests.
A first-principles analytical modeling framework for ML systems.
Designed for education and early design-space reasoning before empirical benchmarking.
pip install mlsysim · Documentation · Getting Started · Issues
mlsysim implements a "Progressive Lowering" architecture, separating high-level workloads from the physical infrastructure that executes them.
| Layer | Domain | Key Components |
|---|---|---|
| Layer A | Workload Representationmlsysim.models |
FLOPs, parameters, and intensity. e.g., Models.Language.Llama3_70B, Models.Vision.ResNet50 |
| Layer B | Hardware Registrymlsysim.hardware |
Concrete specs for real-world silicon. e.g., Hardware.Cloud.H100, Jetson, ESP32 |
| Layer C | Infrastructuremlsysim.infrastructure |
Grid profiles and datacenter sustainability. e.g., PUE, Carbon Intensity, WUE |
| Layer D | Systems & Topologymlsysim.systems |
Nodes, racks, fleet configurations, and network fabrics. e.g., Systems.Racks.DGX_H100_4Node, Systems.Clusters.Frontier_8K |
| Layer E | Execution & Resolversmlsysim.engine.solver |
The 3-tier math engine: Models, Solvers, and Optimizers (Design space search). |
Scenarios.* sits above the stack as the runnable composition layer: it pairs a
Models.* workload with a Hardware.* or Systems.* target and local
constraints such as latency or power. Non-executable real-world anchors live in
ReferenceStats.*. There are no compatibility aliases between these namespaces:
new and existing code should use the canonical path directly.
mlsysim is a first-principles analytical modeling framework for ML systems. It provides a terminal UI for humans and strict JSON output for scripts, CI/CD pipelines, and validation tooling.
Accuracy note: Trust mlsysim for bottleneck classification and relative comparisons. Absolute latency is workload-dependent; well-calibrated cases are often within ±15–30%, while production serving can be 1.5–2× slower than idealized roofline bounds. For production capacity planning, validate with benchmarks.
Discover built-in hardware, models, and infrastructure without reading source code:
mlsysim zoo hardware
mlsysim zoo models
Evaluate the physics of a workload on a specific hardware node instantly: mlsysim eval Llama3_8B H100 --batch-size 32
Define your entire cluster and SLA constraints in a declarative mlsys.yaml file:
# example_cluster.yaml
version: "1.0"
name: "Llama-3 70B training audit"
workload:
name: "Llama3_70B"
batch_size: 4096
hardware:
name: "H100"
accelerators: 64
ops:
region: "Quebec"
duration_days: 14.0
constraints:
assert:
- metric: "performance.latency"
max: 50.0Then compile and evaluate the 3-lens scorecard (Feasibility, Performance, Macro): mlsysim eval example_cluster.yaml
Every command supports strict, schema-validated JSON output. If an assert constraint is violated, the CLI returns a semantic Exit Code 3.
# Export the JSON Schema for your IDE or validation tooling
mlsysim schema > schema.json
# Run an evaluation in a CI pipeline
tco=$(mlsysim --output json eval example_cluster.yaml | jq .m_tco_usd)Use the Tier 3 Engineering Engine to automatically find the optimal configuration:
mlsysim optimize parallelism example_cluster.yaml
mlsysim optimize placement example_cluster.yaml --carbon-tax 150
Because this core powers a printed textbook, we enforce strict Invariant Verification. Registry constants are traceable to primary sources where available, and dimensional integrity is enforced via pint.
TrainingMemoryModel: weights, gradients, optimizer state, activations, and communication buffers per accelerator.ServingCapacityModel: first-pass replica sizing from QPS, target P99 latency, generated length, batching capacity, and queueing.MoERoutingModel: MoE active-parameter and expert-parallel traffic sensitivity under hot-expert imbalance.
MLSys·im is an analytical modeling framework for first-pass reasoning, not a production serving or orchestration system. The 22 walls model physical and economic constraints that bound ML system performance. Several critical production concerns are deliberately out of scope:
| Concern | Why it matters | Where to learn more |
|---|---|---|
| Data drift / distribution shift | The #1 cause of production ML failures — model accuracy degrades silently as input distributions change | Sculley et al. (2015), "Hidden Technical Debt in ML Systems" |
| Model versioning & rollback | Production requires running multiple versions, A/B testing, and safe rollback | Huyen (2022), Designing Machine Learning Systems |
| Monitoring & observability | You cannot manage what you cannot measure — prediction distributions, latency percentiles, error rates | Google SRE Book (2016); Huyen (2022) |
| Feature store freshness | Stale features silently degrade real-time models (recommendations, fraud detection) | Uber Michelangelo (2017) |
| Software bugs & misconfigurations | Most outages are caused by software, not hardware | Barroso et al. (2018) |
| Human factors | Team velocity, on-call burden, and organizational alignment often dominate outcomes | Brooks (1975), The Mythical Man-Month |
Passing all 22 walls is necessary but not sufficient for a successful production deployment.
Students using this tool should understand that infrastructure physics (what mlsysim models) is one dimension of a multi-dimensional engineering challenge.
If you use mlsysim in your research or teaching, please cite:
@software{mlsysim2026,
author = {Janapa Reddi, Vijay},
title = {{MLSys$\cdot$im}: First-Principles Infrastructure Modeling for Machine Learning Systems},
year = {2026},
url = {https://mlsysbook.ai/mlsysim},
version = {0.1.2},
institution = {Harvard University}
}MLSys·im is designed to be highly modular. Install only what you need:
# Core physics engine only (fastest, smallest footprint)
pip install mlsysim
# The CLI and YAML support are included in the base package.
# The [cli] extra is retained as a backward-compatible no-op.
pip install "mlsysim[cli]"
# Install plotting dependencies
pip install "mlsysim[viz]"The framework is just as useful inside a Python script or Jupyter Notebook. The SystemEvaluator provides a clean, unified entry point for full-stack analysis:
import mlsysim
from mlsysim.engine.evaluation import SystemEvaluator
# 1. Define the scenario
model = mlsysim.Models.Language.Llama3_8B
hardware = mlsysim.Hardware.Cloud.H100
# 2. Run the evaluation
evaluation = SystemEvaluator.evaluate(
scenario_name="Llama-3 8B on H100",
model_obj=model,
hardware_obj=hardware,
batch_size=32,
precision="fp16",
efficiency=0.45
)
# 3. View the formatted scorecard
print(evaluation.scorecard())The efficiency parameter (0.0–1.0) captures the gap between peak hardware performance and what your software stack actually achieves. Use these guidelines:
| Scenario | Efficiency | Rationale |
|---|---|---|
| Training (Megatron-LM, large Transformer) | 0.40–0.55 | Well-optimized GEMM + FlashAttention |
| Training (PyTorch eager, small model) | 0.08–0.15 | Kernel launch overhead dominates |
| Inference decode, batch=1 | 0.01–0.05 | Memory-bound; compute nearly idle |
| Inference decode, batch=32+ | 0.15–0.35 | Batch amortizes weight loading |
| Inference prefill, long context | 0.30–0.50 | Compute-bound GEMM + attention |
| TinyML (TFLite Micro on ESP32) | 0.05–0.15 | Interpreter overhead, no tensor cores |
Thanks to these wonderful people for helping improve MLSys·im!
Legend: Bug Hunter · Code Contributor · Documentation Contributor · Design Contributor · Idea Contributor · Code Reviewer · Test Engineer · Tool Builder
 Vijay Janapa Reddi 🧑💻 🎨 ✍️ 🧠 maintenance |
 Rocky 🪲 🧑💻 🎨 ✍️ |
 Farhan Asghar 🪲 🧑💻 🎨 ✍️ |
 Peter Koellner 🪲 ✍️ |
 Zeljko Hrcek 🧑💻 |
Recognize a contributor: Comment on any issue or PR:
@all-contributors please add @username for code, doc, ideas, or bug
Code: Apache License 2.0 — free for commercial and non-commercial use, with patent grant and attribution requirement.
Documentation and textbook prose: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 (CC-BY-NC-SA-4.0) — the tutorials and prose on mlsysbook.ai/mlsysim are part of the Machine Learning Systems textbook and carry its license.
The two licenses are intentionally separate: the Python package is permissively licensed so engineers and researchers can use it anywhere (including commercially), while the textbook prose retains its non-commercial protection to prevent republication as a derivative textbook.
Copyright © 2026 Vijay Janapa Reddi and MLSys·im contributors.