RFC: OpenAI-compatible server — mlx_lm.server already serves 35/40 caches, but compression is accounting-only at runtime

[rajveer43]

Summary

Proposal under discussion: ship an OpenAI-compatible HTTP wrapper inside VeloxQuant-MLX so users can integrate via the OpenAI SDK instead of the 3-line KVCacheConfig → KVCacheBuilder → model.make_cache pattern.

I ran a feasibility pass against v0.41.0 / mlx_lm 0.31.3. Two findings invert the premise, and I'd like to settle them before any server code is written:

1. mlx_lm.server already serves our caches today. 35/40 methods subclass mlx_lm.models.cache.KVCache and inherit the whole serving contract. A "wrapper" is ~10 lines, not a new subsystem.
2. But the compression is accounting-only at runtime. Caches store dequantized fp16 tensors, so the inherited nbytes over-reports by ~4× vs fp16-raw (and ~16–32× vs true compressed size), and no memory is actually saved.

A server shipped on top of (2) would serve confident, wrong numbers under a banner that implies memory relief. That's a credibility risk to the benchmark work that is this project's core asset — hence this issue rather than a PR.

Verdict: build-differently. Fix the memory/accounting story first; the server is a ~10-line follow-on.

Everything below was executed, not inferred. Where I'm inferring or don't know, I say so.

Where

Finding 1 — the caches already satisfy the serving contract

mlx_lm.models.cache.make_prompt_cache defers to model.make_cache() when present (models/cache.py:31-32) — exactly the hook patch_model_kv_cache overrides. Runtime classification of all 40 methods:

SUBCLASS mlx_lm KVCache: 35   (turboquant_rvq, kivi, vecinfer, svdq, palu, xquant, h2o, snapkv, …)
STANDALONE:              5    (turboquant_prod, turboquant_mse, polar, qjl, spectral)

Those 35 inherit update_and_fetch, nbytes, trim, is_trimmable, merge, meta_state for free, e.g. veloxquant_mlx/cache/turboquant_rvq_cache.py:32:

from mlx_lm.models.cache import KVCache as _MLXKVCache

class TurboQuantRVQKVCache(_MLXKVCache):

Correcting my own earlier note: a static grep for def merge / def trim returns 0 across all 45 cache files, which looks like a total gap. That's wrong — those methods are inherited, not defined. Every merge/trim grep hit was prose in docstrings or config field names (cam_merge, minicache_*). Worth flagging so nobody re-derives the same false conclusion.

Finding 2 — compression is bookkeeping alongside uncompressed storage

veloxquant_mlx/cache/turboquant_rvq_cache.py:69-89 encodes, immediately decodes, increments two byte counters, then stores the dequantized fp16 tensor:

ev        = self._quantizer.encode(k_unit)
k_hat_u   = self._quantizer.decode(ev)
k_dequant = (k_hat_u.astype(kdtype) * safe).reshape(B, H, S, D)   # line 82

per_tok = (math.ceil(self._head_dim * 2 * self._bits / 8) + 2) * H * B
self._key_bytes_compressed += per_tok * S                          # line 86  <- counter only
self._key_bytes_fp16       += H * B * S * self._head_dim * 2

return super().update_and_fetch(k_dequant, values)                 # line 89  <- stores fp16

Measured on a live cache (S=64, H=8, D=128, bit_width_inlier=2):

turboquant_rvq (TurboQuantRVQKVCache)
   keys dtype float16, shape (1, 8, 256, 128)   <- full fp16 tensor retained
   nbytes = 1048576   (fp16 raw would be 262144)

Same result for kivi and vecinfer.

To be clear: this is entirely defensible for the library's actual purpose. Measuring fidelity/compression trade-offs is what compressed_key_bytes / fp16_key_bytes (turboquant_rvq_cache.py:91-99) exist for, and the README is careful about it. It only becomes a problem under a server, which implicitly promises memory relief.

Why this breaks under a server

mlx_lm.server runs on ThreadingHTTPServer (server.py:1706). Traced do_POST → fetch_nearest_cache → insert_cache:

1. Wrong byte accounting (silent, highest severity). insert_cache computes sum(c.nbytes for c in prompt_cache) unguarded (models/cache.py:1704). Inherited nbytes returns keys.nbytes + values.nbytes over dequantized tensors → measured 1048576 vs 262144 fp16-raw. LRU eviction and --prompt-cache-bytes budgeting are therefore driven by fiction; the server evicts far too aggressively. No exception, no warning.

2. The 5 standalone methods hard-crash — including our default turboquant_prod:

turboquant_prod: AttributeError: 'TurboQuantKVCache' object has no attribute 'is_trimmable'
turboquant_mse:  AttributeError: 'TurboQuantKVCache' object has no attribute 'is_trimmable'
polar:           AttributeError: 'PolarQuantKVCache' object has no attribute 'is_trimmable'
qjl:             AttributeError: 'QJLKVCache' object has no attribute 'is_trimmable'
spectral:        AttributeError: 'SpectralQuantKVCache' object has no attribute 'is_trimmable'

3. Cross-request contamination does NOT occur (I expected it to). patch_model_kv_cache pins one shared cache list via a closure, but fetch_nearest_cache returns copy.deepcopy(cache_entry.prompt_cache) (models/cache.py:1676), and deepcopy succeeds on every compressed cache tested. Each request gets its own copy. Ruling this out.

4. Compression vs prefix reuse — not currently a conflict, precisely because compression isn't real at runtime. It becomes live the moment we fix storage: genuinely compressed state must still survive deepcopy and trim. This is a design constraint on the fix, not a blocker.

Options considered

Option	Eng. cost	Maintenance vs upstream drift	Blast radius	Capability unlocked
(a) Thin launcher (veloxquant serve → mlx_lm.server)	~10 lines	Low	None	OpenAI API, wrong nbytes, no real memory saving
(b) Own server in veloxquant_mlx/serve/	~1900 lines duplicated	High	Medium	Nothing (a) doesn't already give
(c) Upstream contribution to make caches pluggable	Medium	Lowest long-term	None	Moot — the make_cache hook already exists
(d) Fix compressed storage + honest nbytes	Medium–high	Low	High (core)	Real memory savings; every MLX serving stack benefits

Recommendation: (d), then (a). (c) is moot — there's nothing to make pluggable. (b) is strictly dominated by (a).

Strongest argument against my own recommendation: (d) touches the core of a library whose credibility rests on 1417 passing tests and published benchmarks — risking the numbers that are the product, to chase a memory benefit no user has yet asked for. A coherent alternative is to ship (a) now with explicitly documented "no runtime memory reduction," and let demand decide. I don't recommend it (a server is exactly where users assume the memory claim without reading caveats), but it's a legitimate call and I'd like input.

Tiered support matrix

Criterion: passes can_trim_prompt_cache, survives deepcopy, reports honest nbytes.

Tier	Count	Methods	Status
Serves, honest bytes	0	—	requires (d)
Serves, over-reports bytes	35	rvq, kivi, vecinfer, svdq, palu, xquant, h2o, snapkv, …	works today; misleading
Crashes	5	turboquant_prod (default), turboquant_mse, polar, qjl, spectral	needs adapter

Whatever ships, the docs should carry a matrix like this — never "41 methods."

Cheapest experiment to run first (~1 afternoon)

Before committing to (d), this is the highest-information thing available:

model, tok = mlx_lm.load("mlx-community/Mistral-7B-Instruct-v0.3-4bit")
patch_model_kv_cache(model, KVCacheConfig(method="turboquant_rvq", bit_width_inlier=1))
# hand off to mlx_lm.server's main()

1. Point the OpenAI SDK at it — confirm end-to-end works (expected: yes).
2. Measure peak RSS vs baseline mlx_lm.server. This is a genuine falsifier of my analysis: if RSS does drop, my central finding is wrong and (d) shrinks dramatically.
3. Show ~5 target users (Ollama/LM Studio-shaped, Apple Silicon); watch whether they assume memory savings unprompted.

Step 2 is the one I most want run, because I have not measured it on a real model (see caveats).

If we proceed — minimum honest v1

Scope: (d) real compressed storage + truthful nbytes for a small tier (turboquant_rvq, kivi, vecinfer) → adapter shim for the 5 standalone caches → veloxquant serve launcher → /v1/kv/stats exposing measured compression.

/v1/kv/stats is arguably the differentiator — compressed_key_bytes/fp16_key_bytes already exist per-cache, and exposing real compression telemetry is the only reason to pick this over stock mlx_lm.server.

Config: launch-time flags, one method per process (--method turboquant_rvq --bits 1). Rejected extra_body (per-request method changes can't rebuild a warm cache) and model@method strings (breaks client model-list assumptions). Calibration (spectral, artifact-backed) runs eagerly at startup, failing loudly — lazy warm-up puts a ~5s stall inside a request that looks like a hang.

Anti-goals: multi-tenancy, auth, rate limiting, Docker, non-Apple platforms, embeddings/fine-tuning endpoints, all 41 methods, custom protocol code.

Acceptance criteria:

• OpenAI Python SDK, unmodified, streaming + non-streaming.
• sum(c.nbytes) within ±5% of measured compressed bytes (today: +300% or worse).
• Peak RSS at fixed context strictly below baseline mlx_lm.server — the claim a server implies. Fails today.
• No unbounded growth over N=100 sequential requests.
• tokens/sec reported against baseline, including regressions.
• Every method either serves correctly or refuses at startup. No silent fp16 fallback.

Dependencies: zero new runtime deps — mlx_lm is already present and http.server is stdlib. The 3-dep footprint holds; no [serve] extra needed under (a).

Risks

Risk	Likelihood	Impact	Structural mitigation
Server implies memory savings that don't exist	Certain today	Severe — hits benchmark credibility	Make (d) a precondition for shipping; assert RSS in CI, not in docs
Wrong nbytes misdrives LRU eviction	Certain	High, silent	Override nbytes to report true compressed bytes; unit-test against _key_bytes_compressed
Default method crashes under server	Certain	High	Startup validation refusing unsupported methods
merge semantics wrong under batching	Unknown	High	Test before relying on inherited merge; else disable batching explicitly
(d) destabilizes core caches	Medium	High	Tier rollout; existing 1417 tests as the gate
Scope creep (auth, Docker, triage)	High	Medium	Anti-goals above, stated in README

Open questions for discussion

1. Can compressed state be stored without dequantizing, while keeping deepcopy and trim working? → Blocks all of (d); determines whether a truthful server is possible at all. Highest priority.
2. Is dequantize-on-store deliberate (fidelity measurement) or incidental? → Decides whether (d) is a fix or a second storage mode alongside the benchmark path. Only @rajveer43 can answer this.
3. Does inherited BatchKVCache.merge produce correct results on compressed caches? → Blocks concurrent batching. Untested.
4. Do the 5 standalone caches get an adapter, or drop out of serving? → Affects the default-method decision (turboquant_prod is in this group).
5. Is there appetite to own a server long-term (auth, deployment docs, triage)? → Blocks (a) regardless of (d).

Caveats on this analysis

Executed: the 35/40 classification; the 5 crashes; nbytes = 1048576 vs 262144 fp16-raw; fp16 storage post-dequantize; deepcopy success; ThreadingHTTPServer; unguarded insert_cache.

Inferred, not observed: LRU over-eviction follows from over-reported nbytes — mechanism traced through the source, not observed end-to-end on a running server.

Not measured: peak RSS on a real model. The "no memory saving" claim is strongly implied by fp16 storage but unverified at model scale — that's the experiment above, and it can falsify the core finding.

Unknown: merge semantics under compression; whether dequantize-on-store is intentional.

Speculation (low confidence, flagged): that integration effort isn't actually the top adoption barrier — discovery, trust in numbers, and 41-method selection paralysis are plausible alternatives. I have no usage data; step 3 of the experiment is the cheap resolution.

[Som23Git]

Follow-on: control panel UX

Filed an oMLX-like Start/Stop control panel epic that treats this RFC as the backend dependency:

• Parent: UX: VeloxQuant control panel (oMLX-like Start/Stop desktop experience) #33
• MVP Start/Stop UI: UI: Server lifecycle — Start/Stop, network, copyable OpenAI endpoints #34
• Method education panel: UI: Method & default-profile panel — understand and pick KV compression without Python #35
• Live KV telemetry (consumes the /v1/kv/stats idea from this RFC): UI: Live KV telemetry & logs — compression, memory, request activity #36

UI acceptance mirrors this issue's honesty rules (no silent fp16 fallback; accounting-only mode must be labeled until option (d) lands).

[rajveer43]

@Som23Git Let me know your thoughts also in this.

[rajveer43]

#37 implements option (a) — the thin launcher — and adds a runtime registry that reproduces this issue's support matrix by probe rather than by hand.

Findings 1 and 2 both confirmed independently. Probing all 40 methods (subclass check → live update_and_fetch → can_trim_prompt_cache → deepcopy) reproduces the 35/5 split exactly, with the same five standalone methods and the same root cause.

SUBCLASS mlx_lm KVCache: 35
STANDALONE:              5   turboquant_prod, turboquant_mse, polar, qjl, spectral

A test asserts this split, so drift fails CI instead of quietly misleading the UI. A second test asserts that no method may claim honest bytes until real compressed storage lands — so option (d) is the only thing that can flip that flag.

On (d)-first vs (a)-first: #37 proposes (a) first, inverting this issue's recommendation. The argument is that blocking the adoption surface on (d) makes it wait for the highest-risk change in the library, while the credibility risk (d) protects against is addressable where the claim is made — hence the permanent accounting-only banner in every surface showing bytes, plus the test above. Reasonable to disagree; the launcher and registry stand alone if you'd rather hold the panel.

The ~10-line estimate was optimistic. Three integration issues surfaced only against a live model:

1. ModelProvider reads ~23 fields off cli_args, so a hand-written namespace breaks on the next upstream option. feat(ui): VeloxQuant control panel — method registry, OpenAI-compatible serve, Start/Stop web UI #37 harvests mlx_lm's own parser defaults instead.
2. MLX arrays are bound to the thread that created them. Loading the model on the main thread and generating on mlx_lm's worker fails with There is no Stream(gpu, N) in current thread. Upstream sidesteps this by calling load_default() inside _generate; feat(ui): VeloxQuant control panel — method registry, OpenAI-compatible serve, Start/Stop web UI #37 hooks _load to land on the same thread.
3. is_batchable is computed at the end of _load. Overriding the load without recomputing it silently routes requests to _serve_single, which generates on the HTTP thread with no stream scope — same crash, different route.

Worth recording here so nobody re-derives them. Details in docs/control-panel-test-plan.md.

Not addressed by #37 and still open for this issue: /v1/kv/stats, honest nbytes, and the 5-method adapter.