MojoServe

High-performance AI inference engine powered by Mojo.

MojoServe is a lightweight inference-serving platform with a Python-first developer experience and a Mojo execution core. The public API should feel as simple as FastAPI or Uvicorn, while the runtime path is designed to move model execution, batching, scheduling, tensor operations, and profiling into Mojo.

pip install mojoserve
mojoserve serve model.onnx
mojoserve benchmark model.onnx

from mojoserve import InferenceServer

server = InferenceServer.from_model("model.onnx")
result = server.predict({"input": [1, 2, 3, 4]})
print(result)

Expected response shape:

{
  "output": [...],
  "latency_ms": 2.1,
  "backend": "mojo"
}

Status

This repository is an early production scaffold. It includes:

• src/mojoserve: Python facade, CLI, FastAPI app, schemas, config, wrappers, client, tests.
• mojo/mojoserve_core: Mojo runtime package boundaries for engine, batching, scheduling, tensor ops, memory, profiling, and adapters.
• mojo/mojoserve_pkg: planned Python extension bridge exporting mojoserve_native.
• docs/architecture.md: detailed system design and implementation order.
• docker, deployment, configs, benchmarks, examples: operational starting points.

The current Python fallback backend exists to keep API contracts testable before the native Mojo extension is compiled. ONNX Runtime is supported as the practical MVP execution bridge while Mojo-native ONNX execution matures.

Development

Pixi is the recommended environment manager for Mojo projects:

pixi install
pixi run test
pixi run lint

Mojo does not run natively on Windows. Use WSL2 or a Linux/macOS environment for Mojo builds.

Python-only development:

python -m venv .venv
.venv\Scripts\activate
pip install -e ".[dev,onnx]"
pytest

CLI

mojoserve serve model.onnx --host 127.0.0.1 --port 8000
mojoserve benchmark model.onnx --requests 100
mojoserve info

HTTP

curl -X POST http://127.0.0.1:8000/predict \
  -H "Content-Type: application/json" \
  -d "{\"input\": [1, 2, 3, 4]}"

Architecture

Python owns user ergonomics: CLI, HTTP, validation, config, logging, lifecycle, schemas, and client SDK.

Mojo owns performance-critical runtime work: model execution, tensor operations, memory layout, batching, scheduling, preprocessing, postprocessing, profiling, and future GPU kernels.

Read docs/architecture.md for the full design.