MiniFlow Serving (v2.0)

Production ML serving layer built on top of MiniFlow (P1a). Adds a FastAPI /predict API, PSI-based drift detection, and a two-proportion z-test A/B testing framework, all instrumented for Prometheus and visualised in Grafana.

Client → /predict → A/B assign (sticky) → model.predict → drift.record (bg) → metrics → JSON
                                                              ↑
                          background loop every 60s → drift.compute_psi → PSI gauge / alert

Quick start (Docker)

docker compose up --build

Four containers come up: api (8000), redis (6379), prometheus (9090), grafana (3000, anonymous admin).

# Prediction
curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"features": [1.2, 0.5, 3.1], "user_id": "user_123"}'
# -> {"prediction": 0.87, "model_variant": "B", "latency_ms": 4.2, "request_id": "a1b2c3d4"}

# Drift + A/B demo (watch Grafana at http://localhost:3000 → "MiniFlow Serving")
python scripts/simulate_drift.py
curl http://localhost:8000/health      # drift_alert flips true within ~60s
curl http://localhost:8000/ab/results  # variant B wins

Endpoints

Method	Path	Purpose
POST	/predict	Inference + sticky A/B variant + drift recording
POST	/convert	Record an A/B conversion for a user
GET	/ab/results	Two-proportion z-test results
GET	/health	Readiness (200 unless models failed to load)
GET	/metrics	Prometheus exposition

Local development

The .venv is preconfigured. Run everything through it:

# Tests (no Redis needed — uses fakeredis / in-memory fallback)
./.venv/Scripts/python.exe -m pytest tests/ -v --cov=app --cov-report=term-missing

# Run the API locally (Redis optional; falls back to in-memory store)
./.venv/Scripts/python.exe -m uvicorn app.main:app --reload

# Load test (requires the API running) — target: p99 < 100ms @ 50 concurrent
./.venv/Scripts/python.exe scripts/load_test.py --requests 1000 --concurrency 50

Design notes

• Sticky, deterministic A/B: variant is decided by md5(user_id) % 10000, so the same user always lands in the same arm even if Redis is wiped. Redis is a cache, not the source of truth.
• Fixed PSI bins: bin edges are computed once from the reference (training) distribution and never change; outer edges are ±inf so tail drift is counted. PSI is computed on a background loop, never on the request hot path.
• Graceful degradation: no Redis → in-memory store (/health reports redis: degraded, still 200); no miniflow registry / registered model → built-in LinearModel fallbacks (different per variant).
• Single worker in the container so the in-process Prometheus registry, model cache, and drift gauge stay consistent. Scale out with more containers behind a load balancer.
• Sync def endpoints, not async def: the store uses the synchronous redis-py client. Declaring the I/O-bound endpoints async would block the event loop on every Redis round-trip and serialise concurrent requests; as plain def Starlette runs them in a threadpool (sized via THREADPOOL_SIZE, default 128).

Verified performance

Measured against the Docker stack (single worker, real Redis):

Metric	Result
In-handler latency (Prometheus histogram)	p99 ≈ 9 ms, avg ≈ 0.7 ms
End-to-end @ 50 concurrent (hey, in-network)	p99 ≈ 94 ms, 1280 req/s
End-to-end @ 50 concurrent (load_test.py, host)	p99 ≈ 88 ms, 673 req/s

Measurement caveat: a single-process asyncio HTTP client is GIL-bound and tops out at a few hundred req/s, which makes the client the bottleneck. load_test.py spreads load across processes; for the truest number use a native tool from inside the compose network (bypasses the Docker Desktop localhost proxy):

docker run --rm --network miniflow-serving_default williamyeh/hey \
  -n 5000 -c 50 -m POST -T application/json \
  -d '{"features":[1,2,3],"user_id":"u"}' http://api:8000/predict

Optional: real registered models

To serve versioned artifacts from the miniflow ModelRegistry instead of the fallbacks:

pip install -e ../MiniFlow      # install P1a
python scripts/seed_models.py   # registers model_v1 (A) and model_v2 (B)