PLAN.md

Kairu — Project Roadmap

流 · to flow, to stream

Current version: v0.29.0

---

Researched Feature Roadmap

A curated map of the next two quarters of work, gathered from internal priorities + outside-in survey of regulated-AI evaluation tooling. Items are scoped by both impact (how much it unlocks for paying users) and complexity (engineering hours + ongoing maintenance). Every P1 item has shipped in v0.15.0 — the four 🔴 rows below are now DONE.

🔴 P1 — Critical (shipped in v0.15.0)

• Score distributions + percentile benchmarks (DONE — v0.15.0) Each criterion now carries a reference distribution built from a deterministic 1000-pair synthetic corpus. GET /benchmarks/{criterion} returns p25/p50/p75/p90/p99/mean/stdev plus a 20-bucket histogram for violin / sparkline rendering. Every /evaluate response includes a benchmarks block mapping each criterion to {you, rank, p25, p50, p75}.
• A/B statistical significance testing (DONE — v0.15.0) POST /compare now returns a significance block with the paired t-test result over per-criterion differences: n, mean_diff, stdev_diff, t_stat, df, p_value, effect_size, effect_label, confidence_interval, winner. A statistical_winner field overrides the heuristic winner to "tie" whenever p > 0.05 or |Cohen's d| < 0.2. Pure stdlib — Student's t CDF via Simpson's rule numerical integration of the PDF.
• Immutable audit log (DONE — v0.15.0) Append-only SQLite at KAIRU_AUDIT_DB (default :memory:). Schema-level triggers RAISE(ABORT, …) on UPDATE / DELETE. Every /evaluate and /compare call records {timestamp_utc, input_hash, rubric_name, rubric_version, judge_model, endpoint, scores, reasoning} and returns the row id in the response. GET /audit?start=&end=&... paginated, GET /audit.csv flat export. WAL journaling — dashboards read while the API writes.
• Rubric versioning (DONE — v0.15.0) Every Rubric carries a SemVer version (default 1.0.0). RUBRIC_REGISTRY[name][version] keeps every version ever served so audit-log rows resolve back to the exact rubric that produced them. POST /rubrics creates a new version (patch-bumps when no version is supplied). GET /rubrics now lists every active version.

🟠 P2 — High Impact / Medium Complexity

• Judge bias correction + multi-judge ensembles (DONE — v0.16.0) JudgeConfig carries (rubric, criteria, weights, seed, noise). Median aggregation per criterion (robust to one outlier), per-criterion stdev as the disagreement metric, disagreement_flag = max_stdev > 0.2 (default; tunable). POST /evaluate/ensemble and POST /compare/ensemble accept judges: [JudgeConfig...]. Deterministic seeded Gaussian noise simulates inter-judge variance authentically while keeping CI reproducible. Real LLM judges plug in behind the same JudgeConfig contract later.

• Production traffic → auto-eval pipeline (DONE — v0.16.0) POST /eval_from_log accepts {input, output, metadata?} records, scores them through a named rubric, and returns a LogEvalReport with mean/median/min/max/stdev aggregates, per-criterion mean and min, and per-item pass/fail flags. passed: bool keys on mean_aggregate >= threshold — CI uses it as the exit-code gate. Metadata passes through untouched for downstream slicing by request id / region / model tag.

• CI regression gating (DONE — v0.16.0 — bonus item) POST /ci/baseline snapshots a golden corpus with BaselineSnapshot (input-hash + per-item scores). POST /ci/check compares a candidate run against the snapshot; a criterion regression is any per-item drop

  threshold (default 0.05). Unmatched-input drift surfaces in the report. FileBaselineStore persists snapshots to KAIRU_CI_DIR as JSON (atomic write via tempfile + rename). Designed for use as a deploy gate.

• Constitutional evals from policy docs (DONE — v0.17.0) POST /rubrics/generate accepts plain-text policy documents and returns auto-generated rubric criteria with weights. NLP extraction of "must", "shall not", "required" clauses → rubric items. The generated rubric registers as a new versioned entry in RUBRIC_REGISTRY.

• Agentic trajectory scoring (DONE — v0.17.0) POST /eval/trajectory accepts a sequence of {step, tool_call, observation, response} records. Scores: tool selection correctness, error recovery, goal progress/completion, efficiency (steps taken vs. optimal). Returns a per-step breakdown plus an overall trajectory grade.

✅ v0.29.0 — Conformal judge intervals: distribution-free uncertainty bounds (DONE)

• kairu/conformal.py — split conformal prediction (Sheng et al., EMNLP 2025, arXiv:2509.18658) adds a distribution-free coverage guarantee to the judge-eval stack, complementing the distribution-parametric reliability metrics. conformal_quantile (finite-sample rank, +inf when undersized), calibrate_interval → ConformalInterval (ordinal boundary clamp + lower-bias midpoint), conformal_from_ensemble bridge. reliability.py / EnsembleResult untouched — purely additive.
• 17 new tests incl. an empirical-coverage check; module 100% coverage.
• First sprint of a new Discovery cycle (researcher sweep surfaced it as High-impact/Low-complexity; rebalances toward eval after two inference sprints). Backlog from the same sweep: IRT judge discrimination, entropy-driven adaptive γ, dark-current datasheet, radix-tree KV dedup.

✅ v0.28.0 — SPEED-Bench task splits + speculative spec/quant warnings (DONE)

• kairu/speed_bench.py — run_speed_bench → SpeedBenchReport runs the benchmark across six semantic task splits (TaskSplit/DEFAULT_SPLITS: translation, summarization, qa, code, dialogue, math) and reports per-split throughput + a throughput coefficient of variation quantifying task-dependence (SPEED-Bench methodology). BenchmarkRunner gained an optional prompt arg to drive distinct splits.
• kairu/auto_profile.py — DecoderProfile.warnings + recommend(quant=, draft_kind=) flag 4-bit draft and tree-draft speculative configs that erode acceptance/speedup (empty unless the hints are supplied).
• 14 new tests; new tests/test_speed_bench.py at 100% module coverage.
• Closes this session's Discovery sweep — both the eval track (CyclicJudge, reliability) and the inference-optimizer track (KV eviction/quant, adaptive early exit, SPEED-Bench) are complete.

✅ v0.27.0 — Adaptive per-token early exit + arch-suitability gating (DONE)

• kairu/early_exit.py — CALM-style (Schuster et al. 2022) per-token adaptive confidence threshold, opt-in via adaptive=True (static path unchanged). The bar decays geometrically from confidence_threshold toward min_confidence over decoding steps via effective_confidence(step); early tokens need high confidence, later tokens exit more readily. Constructor now validates inputs; stats gain adaptive + final_confidence_threshold.
• kairu/auto_profile.py — _early_exit_suitable gates early exit out of encoder-style architectures (BERT/RoBERTa/T5/…) and sub-6-layer models, falling back to vanilla with an explanatory rationale.
• kairu/wrapper.py — adaptive_early_exit flag (parallel to adaptive_gamma) forwards to the decoder from the public entry point.
• 17 new tests; new tests/test_early_exit.py takes the module 21% → 100%.

✅ v0.26.0 — Attention-weighted KV eviction + INT8/INT4 quant tier (DONE)

• kairu/kv_cache.py — upgrades LogitsCache from plain-recency LRU with two opt-in strategies (defaults unchanged, prior path bit-exact):
  • eviction="attention" — H2O heavy-hitter eviction (Zhang et al. 2023): evict the least-accumulated-attention entry, not the oldest. Hits accrue attention; add_attention(key, weight) injects external rollups; ties break on the oldest entry (graceful LRU degradation).
  • quant="int8" | "int4" — affine min-max quantised storage tier, ~4×/~8× footprint reduction at ≤ half-step precision loss; int4 packed two-per-byte. New frozen QuantizedArray type; stats() exposes eviction/quant/ memory_bytes.
• CachedModel forwards both knobs. kairu/__init__.py exports QuantizedArray.
• 18 new tests; tests/test_kv_cache.py at 100% module coverage, all grade A.

✅ v0.25.0 — Psychometric reliability metrics (DONE)

• kairu/reliability.py — cronbach_alpha (internal consistency across criteria), intraclass_correlation (ICC(2,1) inter-judge agreement on continuous scores), fleiss_kappa (chance-corrected agreement on pass/fail binarisation), compute_reliability → ReliabilityReport with standard interpretation bands, reliability_from_ensemble. Pure stdlib; each metric returns None when undefined rather than fabricating a number. Grounded in Autorubric (arXiv:2603.00077).
• API: /evaluate/ensemble responses gain a reliability block; new POST /evaluate/reliability computes from a raw judges × criteria matrix.
• 25 new tests; suite: 802 passed, 4 HF-gated skipped.

✅ v0.24.0 — CyclicJudge: round-robin allocation + coverage-correct intervals (DONE)

• kairu/cyclic_judge.py — cyclic_allocate (round-robin judge assignment, CyclicJudge / arXiv:2603.01865), cyclic_evaluate → CyclicEvalReport (one rotating judge per item at single-judge cost + load-balance diagnostic), batch_mean_interval → MeanInterval (Student-t CI over independent per-item aggregates — the coverage-correct sampling unit; cf. CJE / arXiv:2512.11150, which shows criterion-level CIs have ~0% coverage), variance_components → VarianceComponents (two-way ANOVA judge/item/residual split), full_grid_scores (N·K reference run).
• POST /evaluate/cyclic — boundary-validated round-robin batch endpoint.
• Housekeeping: reconciled the 0.20.0→0.24.0 version-stamp drift across pyproject.toml + kairu/__init__.py; fixed the watermark.py docstring DeprecationWarning.
• 41 new tests; suite: 777 passed, 4 HF-gated skipped.

✅ v0.23.0 — Rubric Marketplace (last P3 item) (DONE)

• kairu/marketplace.py — MarketplaceEntry, MarketplaceStore (SQLite+WAL), compute_signature, open_default_marketplace_store, seed_community_rubrics
• api/marketplace_router.py — GET/POST /marketplace, GET /marketplace/domains, GET /marketplace/{name}, POST /marketplace/{name}/import
• 4 community rubrics seeded at startup: medical_qa, legal_analysis, creative_writing, code_review
• Marketplace tab (18th tab) — domain chips, search, animated rubric cards, one-click import, publish form
• 26 new tests; suite: 640 passed, 4 HF-gated skipped

✅ v0.21.0 — Audit/RubricLab/Batch tabs + Human Feedback + Visual Overhaul (DONE)

• Visual overhaul — animated mesh bg, glassmorphism cards, slide/fade tab transitions, neon glow, sparklines, animated histogram
• Audit tab — live eval history query (GET /audit), CSV export, color-coded scores
• Rubric Lab tab — constitutional rubric generator (POST /rubrics/generate) + rubric browser
• Batch tab — multi-pair batch eval (POST /batch) with animated progress bar
• Human Feedback — kairu/human_feedback.py (FeedbackStore, SQLite), POST /eval/{id}/feedback endpoint, 👍/👎 UI in Evaluate tab
• Generate tab — Prompt Library drawer backed by /prompts API
• Leaderboard — sparkline graphs + rank badge glow
• 604 tests passing, all CI gates green

✅ v0.20.1 — Live Demo UI: Engine / Speed-Up / Watermark + Prism Fix (DONE)

• Engine tab — Speculative Decoder (animated token bubbles), Layerwise Early Exit (depth bars), KV Cache Monitor (hit/miss/evict timeline) — all live via /api/kv-cache-sim, /api/early-exit-sim
• Speed-Up tab — interactive 10×8 speedup heatmap + AutoProfile strategy card via /api/recommend
• Watermark tab — Kirchenbauer token pills + z-score glow display via /api/watermark-demo
• Prism beam animation fixed — stroke-dashoffset proportional to rubric scores (was no-op)
• Decoder hint in Generate tab — calls /api/recommend inline after streaming completes
• 517 tests passing, all gates green

✅ v0.20.0 — Real Leaderboard + Score Analytics + Prompt Library (DONE)

• kairu/leaderboard.py — SQLite-backed score history; rank() with delta/trend/percentiles
• kairu/analytics.py — histogram, nearest-rank percentiles, z-score anomalies (pure stdlib)
• kairu/prompts.py — saved prompt library with tag normalization
• Live leaderboard + analytics in demo UI; "synthesised view" badges removed
• 590 tests passing

🔵 v0.19.0 — Tooling that turns kairu into a service

• Evaluation templates (DONE — v0.19.0) POST /templates saves a named rubric+criteria+weights+judges bundle to SQLite (KAIRU_TEMPLATE_DB). GET /templates, GET /templates/{name}, DELETE /templates/{name} for CRUD. Apply via POST /evaluate/template/{name} — single-mode when no judges are present, ensemble-mode when they are. Eliminates copy-paste of repeated configurations across CI pipelines.
• Adversarial prompt + response detection (DONE — v0.19.0) POST /evaluate/adversarial_check returns {is_adversarial, confidence, risk_level, patterns_found, categories}. 18 default heuristics across five categories (prompt_injection, jailbreak, override, exfiltration, compliance); each pattern has a calibrated weight and targets the prompt, the response, or both. Detects classic DAN/developer-mode/ignore-previous injection attempts, system-prompt leaks, raw secret / PII / private-key exfiltration, and compliance markers (response openly states it dropped its rules). Pure stdlib regex — no ML dependency.
• Multi-model tournament (DONE — v0.19.0) POST /tournament runs round-robin pairwise ensemble_compare across every (model, prompt) cell of a pre-computed response grid. Returns Elo-style ratings (start 1500, K=32, total Elo conserved), win matrix, per-criterion dominance counts, and a ranked leaderboard. Tournaments persist in memory; GET /tournaments/{id} and GET /tournaments.

✅ P3 — Strategic (all shipped)

• Human feedback integration (DONE — v0.21.0) POST /eval/{id}/feedback records human overrides on individual audit rows. FeedbackStore (SQLite) accumulates votes per criterion.
• Cross-model regression testing (DONE — v0.22.0) GET /regression?model_a=&model_b= flags regressions >2% per criterion. Collapsible panel in Leaderboard tab — pick two models, see the diff.
• Rubric marketplace (DONE — v0.23.0) kairu/marketplace.py + api/marketplace_router.py. 4 community rubrics seeded at startup (medical, legal, creative_writing, code_review). Domain filter chips, keyword search, card grid, one-click import, publish form. 18-tab demo UI fully wired to the eval API.

---

Phase 1 — Core Engine (v0.1.0) ✅ COMPLETE

Ship Gate: 31 Python tests passing.

Deliverables:

• ModelInterface abstract base — zero-dependency contract for any model backend
• MockModel — deterministic LCG-seeded mock; enables full test coverage without ML frameworks
• SpeculativeDecoder — draft-model lookahead with acceptance-ratio rejection sampling (Chen et al. 2023)
• EarlyExitDecoder — confidence-threshold + entropy-floor halting
• TokenBudget — hard prompt+generation cap with consume(), remaining, utilization()
• GenerationMetrics — wall-clock timing, tok/s, mean latency, acceptance rate
• KairuDashboard — Rich live panel for real-time metric display
• ModelWrapper + wrap_model() — unified entry point wiring all layers together
• HuggingFaceModel / _hf_backend.py — optional HF integration (behind kairu[hf])
• pyproject.toml with hatchling build, dev + hf extras
• GitHub Actions CI — Python 3.11, pytest -v

---

Phase 2 — HuggingFace Integration (v0.2.0) ✅ COMPLETE

Ship Gate: 51 Python tests passing (4 gated HF integration tests skipped without KAIRU_TEST_HF=1).

Deliverables:

• kairu/tokenizer.py — TokenizerBase ABC, MockTokenizer (deterministic, no deps), HFTokenizer (wraps HF AutoTokenizer)
• kairu/streaming.py — StreamingDecoder: greedy/temperature-sampled token-by-token iterator using only NumPy + ModelInterface; stream() yields IDs, generate() collects to list; stop-token support
• kairu/_hf_backend.py — full rewrite: HuggingFaceModel with encode(), decode(), stream_generate() (HF TextIteratorStreamer + threading); all heavy imports deferred to __init__ so the module is importable without ML libs
• kairu/__init__.py — exports StreamingDecoder, MockTokenizer, TokenizerBase; guarded import of HFTokenizer; version bumped to 0.2.0
• 8 tokenizer tests (tests/test_tokenizer.py) — fully offline
• 8 streaming tests (tests/test_streaming.py) — uses MockModel, no HF
• 8 HF backend tests (tests/test_hf_backend.py) — 4 structural (offline), 4 integration gated behind KAIRU_TEST_HF=1

---

Phase 3 — Benchmarking (v0.3.0) ✅ COMPLETE

Ship Gate: 59 Python tests passing (4 gated HF integration tests skipped without KAIRU_TEST_HF=1).

Deliverables:

• kairu/bench.py — BenchmarkRunner driving N generation runs against any ModelInterface; build_parser() + main() CLI entry; p50/p95/p99/stddev via pure statistics + sorted-list percentile (no scipy)
• kairu/bench.BenchmarkResult — dataclass with latencies_s, p50, p95, p99, mean, stddev, tokens_per_s_mean, hardware, timestamp; to_json() + save() (never overwrites)
• kairu/bench._collect_hardware() — hostname, OS, machine, CPU model (sysctl//proc/cpuinfo), total RAM (psutil/sysctl hw.memsize//proc/meminfo), Python version
• kairu/__main__bench.py — thin re-export shim for build_parser, main, _collect_hardware
• python -m kairu.bench --model mock --tokens 100 --runs 50 --warmup 5 exits 0 with no ML deps
• 8 benchmark tests in tests/test_bench.py — shape, percentile ordering, JSON round-trip, file save, hardware keys, CLI exit 0, CLI --help, filename contains timestamp+name
• kairu/__init__.py exports BenchmarkRunner, BenchmarkResult; version bumped 0.2.0 → 0.3.0

---

Phase 4 — Streaming API (v0.4.0) ✅ COMPLETE

Ship Gate: 73 Python tests passing (4 gated HF integration tests skipped without KAIRU_TEST_HF=1).

Deliverables:

• kairu/server.py — create_app(model?, tokenizer?, config?) FastAPI factory; POST /generate SSE endpoint, GET /health; OpenAI-compatible chat.completion.chunk frames + kairu extension carrying per-token token_id/index/latency_ms/tokens_per_s; final frame's finish_reason ∈ {length, stop, timeout}; trailing data: [DONE]\n\n sentinel
• kairu.server.ServerConfig — max_prompt_chars, max_tokens_cap, request_timeout_s, rate_limit_requests, rate_limit_window_s; every limit enforced at the API boundary before the tokenizer is touched
• kairu.server.RateLimiter — pure-stdlib sliding-window per-key limiter, asyncio.Lock-guarded
• Boundary validation: empty/oversized prompts, control characters, max-tokens cap, temperature ∈ [0, 2], non-positive stop_token_id
• SHA-256-only prompt logging (raw content never logged)
• 14 server tests in tests/test_server.py — health, OpenAI chunk shape, [DONE] sentinel, all validation paths, 429 rate limiting, request timeout, sliding-window unit tests, total_s monotonicity
• pyproject.toml — new server extras (fastapi, uvicorn, pydantic); dev extras gain pytest-asyncio and httpx; asyncio_mode = "auto"
• kairu/__init__.py — guarded re-export of create_app, ServerConfig, RateLimiter; version 0.3.0 → 0.4.0

---

Phase 5 — Model-Aware Optimization (v0.5.0) ✅ COMPLETE

Ship Gate: 112 Python tests passing (4 gated HF integration tests skipped without KAIRU_TEST_HF=1).

Deliverables:

• kairu/layered.py — LayeredModelInterface extension; MockLayeredModel with depth-monotonic confidence; LayerwiseEarlyExitDecoder reporting per-token exit layers and compute_saved
• kairu/kv_cache.py — LogitsCache (bounded LRU, O(1) get/put, hit/miss/evict stats); CachedModel wrapper memoizing next_token_logits keyed by prefix tuple
• kairu/gamma_scheduler.py — DynamicGammaScheduler (AIMD over γ, configurable bounds/thresholds/window/rates)
• kairu/auto_profile.py — AutoProfile.recommend(model, name_hint?, has_draft=False) → frozen DecoderProfile{strategy, gamma, threshold, temperature, use_cache, cache_capacity, rationale}
• kairu/speculative.py — optional scheduler kwarg; per-round scheduler.update; stats now include final_gamma and gamma_adjustments
• kairu/wrapper.py — new flags cache_capacity (transparently wraps target+draft in CachedModel) and adaptive_gamma (auto-attaches scheduler)
• 39 new tests across tests/test_layered.py, tests/test_kv_cache.py, tests/test_gamma_scheduler.py, tests/test_auto_profile.py
• kairu/__init__.py — exports new types; version 0.4.0 → 0.5.0

---

Phase 6 — Distributed & Production Hardening (v0.6.0) ✅ COMPLETE

Ship Gate: 139 Python tests passing (4 gated HF integration tests skipped without KAIRU_TEST_HF=1).

Deliverables:

• kairu/rate_limit.py — RateLimiterBackend protocol; InMemoryBackend (default, BC-preserving) + RedisBackend (atomic MULTI pipeline with speculative-add rollback). kairu.create_app(..., rate_limit_backend=...) accepts any backend.
• kairu/metrics_export.py — pure-stdlib Prometheus exposition; Counter/Gauge/Histogram (canonical Prometheus latency buckets, O(log buckets) observe). MetricsCollector exposes the named series the dashboard contract depends on.
• GET /metrics endpoint on the SSE server, instrumenting /health, /generate success/422/429/500, and the active-streams gauge with proper try/finally.
• kairu/cli.py — kairu serve | bench | version console script. serve covers every ServerConfig field plus --cache-capacity, --adaptive-gamma, --redis URL.
• kairu/_hf_backend.py — new HuggingFaceKVCachedModel; persistent past_key_values keyed by longest-common-prefix between successive calls; kv_cache_stats and reset_cache() exposed.
• Dockerfile (multi-stage slim, non-root uid 1001, healthcheck) + .dockerignore.
• .github/workflows/docker.yml — multi-arch (amd64/arm64) GHCR publish on main push and v*.*.* tags. Buildx + QEMU + GHA cache. Forks safe.
• 27 new tests across tests/test_rate_limit.py (12), tests/test_metrics_export.py (8), tests/test_cli.py (7); plus 2 server-side /metrics tests.
• kairu/__init__.py — exports MetricsCollector, InMemoryBackend, RedisBackend, RateLimiterBackend; version 0.5.0 → 0.6.0. pyproject.toml adds redis extra and kairu console script.

---

Phase 7 — Observability & Real-Workload Validation (v0.7.0) ✅ COMPLETE

Ship Gate: 191 Python tests passing (4 gated HF integration tests skipped without KAIRU_TEST_HF=1).

Deliverables:

• kairu/tracing.py — KairuTracer (OTel API facade + automatic _NoOpTracer fallback when opentelemetry-api is absent). W3C traceparent/tracestate extraction via extract_trace_context(headers). Per-token add_event annotations on the kairu.generate root span (not child spans — avoids trace store bloat). record_generation_complete() / record_error() helpers. headers_from_request() normalises ASGI headers for propagation.
• kairu/cluster_budget.py — ClusterTokenBudget (Redis INCRBY/DECRBY/EXPIRE with speculative-rollback on cap overflow) + LocalClusterBudget (in-process, asyncio.Lock-guarded, window-resetting). Both implement the ClusterBudgetBackend protocol. Configurable scope string for multi-tenant isolation.
• kairu/server.py — JSONL streaming fallback: when the client sets Accept: application/x-ndjson the server emits the same frame objects as newline-delimited JSON (no data: prefix, no [DONE] sentinel). OTel tracing wired into the _token_loop shared generator — SSE and JSONL paths both get per-token span events. create_app() now accepts an optional tracer: KairuTracer kwarg. Server version bumped to 0.7.0.
• benchmarks/run_corpus.py — CorpusBenchmarkRunner driving the full 100-prompt corpus (instruction-following, Q&A, coding, summarisation, free-form) against any ModelInterface. --model mock runs fully offline. Results saved via BenchmarkResult.save() to benchmarks/results/ (never overwrites). Standalone CLI: python benchmarks/run_corpus.py --model mock --tokens 64 --runs 100.
• helm/kairu/ — Helm chart v0.7.0: Chart.yaml, values.yaml (image, replicas, resources, probes, autoscaling, Redis, OTel, ServiceMonitor, PDB toggles), templates (deployment.yaml, service.yaml, configmap.yaml, hpa.yaml, ingress.yaml, servicemonitor.yaml, pdb.yaml, _helpers.tpl).
• kustomize/ — Kustomize base (Deployment + Service) + overlays: production (4 replicas, doubled resources) and staging (1 replica).
• 52 new tests: tests/test_tracing.py (13), tests/test_cluster_budget.py (19), tests/test_jsonl_stream.py (10), tests/test_corpus_bench.py (10). All run offline with MockModel.
• kairu/__init__.py — exports KairuTracer, extract_trace_context, ClusterTokenBudget, LocalClusterBudget; version 0.6.0 → 0.7.0.
• pyproject.toml — version 0.6.0 → 0.7.0; new otel optional extra (opentelemetry-api, opentelemetry-sdk, opentelemetry-exporter-otlp-proto-grpc).

---

Phase 8 — Adaptive Router (v0.8.0) ✅ COMPLETE

Ship Gate: 221 Python tests passing (4 gated HF integration tests skipped without KAIRU_TEST_HF=1).

Deliverables:

• kairu/router.py — DecoderRouter: given prompt token IDs and runtime signals (prompt length, draft-model availability, latency budget), deterministically selects the optimal decoding strategy (streaming / speculative / early_exit). Routing rules: tight budget (< 200 ms) → streaming; short prompt (< threshold) → streaming; draft available → speculative; otherwise → early_exit. EWMA latency tracking per strategy via record_outcome(decision, metrics).
• kairu/feedback.py — FeedbackLoop: ingests BenchmarkResult objects, buffers until min_results reached, then computes mean acceptance rate and drives DynamicGammaScheduler up (high AR > 0.75) or down (low AR < 0.40) via 100 %/0 % synthetic update rounds. Emits FeedbackSummary on each flush cycle.
• RouterDecision dataclass — carries strategy, profile (a DecoderProfile with strategy overridden to match routing), confidence, rationale, latency_budget_ms.
• RoutingStats dataclass — per-strategy decision counts, per-strategy EWMA latency, total routed.
• FeedbackSummary dataclass — n_results, mean_acceptance_rate, gamma_adjusted, new_gamma, recommendation.
• 16 router tests in tests/test_router.py — construction, all routing branches, budget override, stats accumulation, EWMA update, profile strategy alignment.
• 14 feedback tests in tests/test_feedback.py — flush threshold, buffer clearing, gamma direction, summary fields, multi-cycle operation.
• kairu/__init__.py — exports DecoderRouter, RouterDecision, RoutingStats, FeedbackLoop, FeedbackSummary; version 0.7.0 → 0.8.0.

---

Phase 9 — Token Watermarking & Integrity (v0.9.0) ✅ COMPLETE

Ship Gate: 239 Python tests passing (4 gated HF integration tests skipped without KAIRU_TEST_HF=1).

Deliverables:

• kairu/watermark.py — Kirchenbauer et al. (2023) green/red list watermarking scheme implemented with pure NumPy + stdlib (hashlib, math, struct). No ML framework dependency.
  • WatermarkLogitsProcessor — at each decoding step, hash-seeds a green/red partition of the vocabulary using the preceding token (or a context window) then adds a scalar bias δ to all green-list logits before softmax. Supports seeding_scheme ∈ {"single", "context"}. Never mutates the input logits array.
  • WatermarkDetector — given a token sequence + optional prompt prefix, reconstructs the per-step green lists (identical seeding parameters) and counts green tokens. Computes z-score against the Binomial(T, 0.5) null (well-approximated by N for T ≥ 20) and one-sided p-value via math.erfc (no scipy). Returns a frozen WatermarkResult dataclass.
  • WatermarkResult — frozen dataclass carrying num_tokens, num_green, green_fraction, z_score, p_value, decision, threshold.
  • _norm_sf(z) — exact normal survival function via math.erfc; no scipy dependency.
• kairu/streaming.py — StreamingDecoder gains optional watermark: WatermarkLogitsProcessor | None constructor kwarg. When set, process() is called on every logit array before sampling; when None the code path is identical to v0.8.0 (zero overhead).
• 18 new tests in tests/test_watermark.py — covering: seed determinism, seed uniqueness, green-list shape/fraction/reproducibility, processor construction validation, logit bias correctness, immutability of input, shape-mismatch error, empty context, scheme divergence, detector construction, empty sequence error, result fields, watermarked-sequence z-score direction, unwatermarked no false positive, frozen result mutation, _norm_sf edge cases and monotonicity.
• kairu/__init__.py — exports WatermarkLogitsProcessor, WatermarkDetector, WatermarkResult; version 0.8.0 → 0.9.0.
• pyproject.toml — version 0.8.0 → 0.9.0; description updated.

---

Phase 11 — Squish Integration: Quantization-Tier Quality Eval (v0.10.0) ✅ COMPLETE

Ship Gate: 286 Python tests passing (4 gated HF integration tests skipped without KAIRU_TEST_HF=1).

Deliverables:

• kairu/squish_eval.py — pure-stdlib quality rubric for evaluating LLM outputs across quantization tiers (FP16 baseline vs INT8/INT4/INT2 quantized).
  • 5-criterion SquishEvaluator (correctness, fluency, faithfulness, completeness, safety) — every score in [0, 1], aggregate is unweighted mean.
  • quality_degradation_report(...) — per-criterion delta + retention% per tier; tiers ordered by descending bit-width.
  • recommended_quant_tier(report, tolerance) — returns the deepest tier still within (1 - tolerance) * 100% retention.
  • All result types are frozen dataclasses with as_dict() for JSON transport.
• POST /compare/quantization in kairu/server.py — Pydantic-validated, rate-limited, pure-CPU endpoint that wraps the module-level functions and returns {report, recommended_tier, tolerance}.
• demo/sample_comparisons/04_quantization_comparison.json — realistic GPT-4 vs INT8/INT4/INT2 sample over 5 prompts with verified retention numbers and ready-to-POST request body.
• 20 new tests (tests/test_squish_eval.py + tests/test_server.py) covering identical/degraded/garbage tiers, empty inputs, length mismatches, recommendation thresholds, frozen-dataclass invariants, JSON round-trip, and HTTP-level integration.
• kairu/__init__.py and pyproject.toml: version 0.9.0 → 0.10.0; description updated.

---

Phase 12 — Evaluation API & A/B Comparison (v0.11.0) ✅ COMPLETE

Ship Gate: 294 Python tests passing, 4 skipped HF-gated (32 evaluation + 16 HTTP-boundary tests added).

Deliverables:

• kairu/evaluation.py — seven heuristic scorers (relevance F1, coherence trigram-uniqueness, conciseness Gaussian, safety regex categories, fluency sentence-length + TTR, specificity entity density, completeness recall). All deterministic, pure-stdlib, bounded to [0, 1].
• Five built-in rubrics: default, helpfulness, safety_focused, concise_qa, creative — composable with weights={…} overrides per call.
• compare() returns Comparison with absolute margin and per-criterion winner using a TIE_EPSILON = 0.005 noise floor.
• evaluate_batch() + to_csv() — batch driver returning JSON or CSV-ready rows.
• api/main.py — FastAPI app exposing POST /evaluate, POST /compare, GET /rubrics, POST /batch, GET /health; pydantic v2 models, 413 on oversize, 422 on bad rubric/criterion.
• api/Dockerfile (slim, non-root, $PORT-aware), api/requirements.txt, render.yaml — deployable to Render / Fly / GKE.
• demo/sample_comparisons/ — 3 runnable A/B fixtures with expected_winner for regression validation.
• kairu/__init__.py — re-exports evaluate, compare, evaluate_batch, to_csv, Evaluation, Comparison, Rubric, CRITERIA, RUBRICS; version 0.10.0 → 0.11.0.

---

Phase 13 — Eight Named Rubrics + Prism UI (v0.12.0) ✅ COMPLETE

Ship Gate: 313 tests passing, 4 HF-gated skipped (13 rubric + 6 API-route tests added).

Deliverables:

• kairu/rubrics.py — RUBRIC_DEFS: eight named rubrics (helpfulness, accuracy, safety, coherence, conciseness, creativity, groundedness, tone) with curated weights + canonical hex color per rubric.
• kairu/evaluation.RUBRICS auto-materialises from RUBRIC_DEFS.
• API: GET /rubrics/{name} + POST /evaluate/rubric/{name} (path-param routing).
• demo/server.py — POST /api/prism runs all eight rubrics on one (prompt, response[, response_b]); pure stdlib, 16 KB input cap.
• demo/index.html — full rebuild as the prism UI: pure dark #06060f, idly-rotating SVG triangular prism, eight color-coded beams, A/B mode with offset dashed beams, click-to-evaluate, hover tooltips, kbd shortcut.
• 19 new tests across tests/test_rubrics.py and api/test_api.py.

---

Phase 14 — Prompt Shield & Content Policy (v0.13.0) ✅ COMPLETE

Ship Gate: 336 Python tests passing, 4 HF-gated skipped (25 new shield tests added).

Deliverables:

• kairu/shield.py — PromptShield, ShieldVerdict (StrEnum: ALLOWED/FLAGGED/BLOCKED), ShieldResult (frozen dataclass), ShieldRule, ShieldConfig, get_default_shield() singleton
• 12 default rules: prompt injection (ignore/disregard/forget/override), jailbreak (DAN mode, developer mode, you-are-now, pretend-no-restrictions), roleplay bypass, prompt manipulation
• PII detection: email, SSN (NNN-NN-NNNN), credit card (16-digit), phone — configurable FLAGGED or BLOCKED
• Fail-open design: shield exceptions log a warning and return ALLOWED (never crash callers)
• Patterns compiled at class construction, not per-call
• Server integration: create_app(shield=...) — shield runs before rate limiting; BLOCKED → HTTP 400 {"error":"blocked","reason":...}; FLAGGED → X-Shield-Warning header, request continues
• CLI: kairu shield "<prompt>" [--json] — uses get_default_shield()
• kairu/__init__.py exports: PromptShield, ShieldConfig, ShieldResult, ShieldVerdict, ShieldRule, get_default_shield
• 25 tests in tests/test_shield.py — all offline except 2 server integration tests (TestClient)

---

Phase 15 — Streaming API via SSE (v0.14.0) ✅ COMPLETE

Ship Gate: 356 Python tests passing, 4 HF-gated skipped (20 new streaming-API tests added).

Deliverables:

• kairu/streaming_api.py — StreamingConfig (dataclass: max_tokens, temperature, stop_sequences, seed, stream_chunk_delay_ms), StreamChunk (frozen dataclass: id, content, finish_reason, index; to_sse_line(), to_dict()), TokenStreamer (wraps StreamingDecoder; stop-sequence detection; max-tokens cap; never raises — exceptions yield finish_reason="error")
• POST /generate/stream endpoint in kairu/server.py — StreamRequest Pydantic model; shield runs synchronously before streaming begins; BLOCKED → HTTP 400 JSON; FLAGGED → X-Shield-Warning header + stream continues; rate-limited; StreamingResponse with media_type="text/event-stream"
• SSE format: data: {chunk_json}\n\n per token, data: [DONE]\n\n sentinel at end; OpenAI-compatible choices[0].delta.content + finish_reason
• Seeded determinism via StreamingConfig.seed — same seed always yields same token sequence
• kairu/__init__.py exports StreamingConfig, StreamChunk, TokenStreamer; version 0.13.0 → 0.14.0
• pyproject.toml version 0.13.0 → 0.14.0
• 20 tests in tests/test_streaming_api.py — unit tests for StreamChunk + TokenStreamer, API endpoint tests (200, content-type, data lines, [DONE], JSON validity, choices field, shared id, shield block, shield flag header, max-tokens enforcement)

---

Phase 16 — Judge Ensemble + CI Regression + Log-to-Eval Pipeline (v0.16.0) ✅ COMPLETE

Ship Gate: 438 Python tests passing, 4 HF-gated skipped (90 new tests across tests/test_ensemble.py [18], tests/test_ci_regression.py [15], tests/test_log_eval.py [11], and 14 HTTP endpoint tests in api/test_api.py).

Deliverables:

• kairu/ensemble.py — JudgeConfig (name, rubric, criteria, weights, seed, noise), JudgeScore, EnsembleResult (median_scores, stdev_scores, median_aggregate, max_disagreement, disagreement_flag), EnsembleComparison. Aggregation uses median per criterion (robust to one outlier) and reports per-criterion stdev as the disagreement metric. Deterministic seeded Gaussian noise simulates inter-judge variance without sacrificing test reproducibility. Real LLM judges plug in behind the same JudgeConfig contract.
• kairu/ci_regression.py — BaselineSnapshot (immutable, JSON-round-trippable), BaselineItem, CriterionRegression, RegressionReport. snapshot_baseline() scores a golden corpus and freezes it. check_against_baseline() matches items by input-hash, flags any per-criterion drop > threshold (default 0.05), reports unmatched-input drift in both directions. BaselineStore (in-memory) + FileBaselineStore (atomic write via tempfile + rename, auto-loads from KAIRU_CI_DIR).
• kairu/log_eval.py — LogItemResult, LogEvalReport. evaluate_log() batch-evaluates {input, output, metadata?} records through a named rubric; returns mean/median/min/max/stdev aggregates, per-criterion mean and min, per-item pass flags, and passed: bool keyed on mean_aggregate >= threshold (default 0.5). Metadata passes through untouched for downstream slicing by request id / region / model tag.
• HTTP endpoints in api/main.py:
  • POST /evaluate/ensemble — single (prompt, response) through N judges
  • POST /compare/ensemble — A/B through N judges with winner + per-criterion breakdown
  • POST /ci/baseline — freeze a golden snapshot; returns snapshot_id
  • POST /ci/check — score a candidate run against a snapshot; returns RegressionReport with passed: bool
  • GET /ci/baselines and GET /ci/baselines/{snapshot_id} — list / inspect snapshots
  • POST /eval_from_log — production-log batch eval gate
• app.state.baselines wires a BaselineStore into create_app (resolvable via KAIRU_CI_DIR env)
• kairu/__init__.py exports: JudgeConfig, JudgeScore, EnsembleResult, EnsembleComparison, ensemble_evaluate, ensemble_compare, judge_evaluate, DEFAULT_DISAGREEMENT_THRESHOLD, BaselineSnapshot, BaselineItem, BaselineStore, FileBaselineStore, CriterionRegression, RegressionReport, snapshot_baseline, check_against_baseline, open_default_store, DEFAULT_REGRESSION_THRESHOLD, LogEvalReport, LogItemResult, evaluate_log, DEFAULT_LOG_THRESHOLD; version 0.15.0 → 0.16.0.
• pyproject.toml version 0.15.0 → 0.16.0; description extended.

---

Phase 19 — Evaluation Templates + Adversarial Detection + Multi-Model Tournament (v0.19.0) ✅ COMPLETE

Ship Gate: 544 Python tests passing, 4 HF-gated skipped (≥60 new test outcomes across tests/test_templates.py [11], tests/test_adversarial.py [17], tests/test_tournament.py [15], and 15 HTTP endpoint tests in api/test_api.py).

Deliverables:

• kairu/templates.py — EvaluationTemplate frozen dataclass + TemplateStore (SQLite, INSERT OR REPLACE semantics, created_utc preserved across updates). Materialises judge configs from stored JSON for ensemble templates. KAIRU_TEMPLATE_DB env resolves the file path (defaults to :memory:).
• kairu/adversarial.py — AdversarialPattern (named regex + category + weight + target), 18 default patterns spanning prompt_injection, jailbreak, override, exfiltration, compliance. check_adversarial(prompt, response) returns AdversarialReport(is_adversarial, confidence ∈ [0,1], risk_level, patterns_found, categories, n_prompt_matches, n_response_matches). Confidence is min(1.0, Σ matched_weight / 2.0); risk-level bands at 0.3 / 0.6.
• kairu/tournament.py — run_tournament(models, prompts, judges) runs every pair × every prompt through ensemble_compare, tracks wins/losses/ties + per-criterion dominance, applies standard chess Elo (start 1500, K=32). TournamentResult carries the win matrix, Elo dict, sorted ModelRanking list. In-memory TournamentStore for retrieval.
• HTTP endpoints in api/main.py:
  • POST /templates, GET /templates, GET /templates/{name}, DELETE /templates/{name}
  • POST /evaluate/template/{name} — applies a saved template (single or ensemble)
  • POST /evaluate/adversarial_check
  • POST /tournament, GET /tournaments, GET /tournaments/{tournament_id}
• kairu/__init__.py exports: EvaluationTemplate, TemplateStore, open_default_template_store, AdversarialPattern, AdversarialMatch, AdversarialReport, ADVERSARIAL_DEFAULT_PATTERNS, check_adversarial, TournamentMatch, ModelRanking, TournamentResult, TournamentStore, run_tournament, DEFAULT_ELO_K, DEFAULT_ELO_START. Version 0.18.0 → 0.19.0.
• pyproject.toml version 0.18.0 → 0.19.0.