Strata-HotRod

78 GB of model. 30 GB of RAM. 12 GB of VRAM. Zero cloud.

A 125-billion-parameter AI on one gaming card, streaming from your SSD.
Nothing leaves your PC.

🔥 hot experts on the card  →  🌡️ warm in RAM  →  ❄️ cold on the SSD

Speed · The app · How it works · Install · Credits

---

🧊 78 GB of model on a 30 GB PC

🔥 50-70 tokens/s on one RTX 3060

🐋 341 GB DeepSeek-V4.1-Flash at about 5 tokens/s

⚡ 0.4 s to pick up a 45K-token conversation

A fork of Strata by Niko1221 and contributors (MIT). Strata keeps Qwen's experts in RAM and reads from the SSD when RAM is short; its docs call that path much slower. This fork works on that path: it spreads the experts over the card, RAM and the SSD and guesses ahead which ones to read, so the 78 GB IQ3_S runs on 30 GB of RAM at 50-70 tokens/s.

Can Qwen3.8-Flash-Next run on an RTX 3060 12 GB with low RAM?

Yes. On one RTX 3060 12 GB with 30 GB of RAM and an NVMe SSD, this fork runs Qwen3.8-Flash-Next (Qwen 3.8 Flash Next, 125B) as the 78 GB IQ3_S at 50-70 tokens/s (131K context; 262K fits). The experts that do not fit in RAM are read from the SSD, and a look-ahead guesses ahead which ones. Measured on one PC: see How fast and the "Not tested" list at the bottom.

How much RAM does it need? This fork was measured on 30 GB. Strata's setup recommends 64 GB for IQ3_S and says that with less RAM it reads the rest from the SSD, much slower.

Does it need a cloud service? No. Everything runs on your PC. Setup downloads the model once from Hugging Face.

Tested on one machine, with one model. Other cards, AMD, Windows, macOS and other model sizes may work in upstream Strata; we have not run them, so we do not describe them here.

How fast

Measured on the machine below; a token is about ¾ of a word. Every answer in these runs was read and checked.

Answer	tokens/s (24 Oct, 131K context)
Code (a Python function, 500-650 tokens)	52 - 55
Reasoning (a two-train problem, ~1,000 tokens)	66 - 71
Story (a paragraph, 100-150 tokens)	50 - 53

Long prompts (262K context, 20 Oct)	Read in	Answer written at
8,554 tokens	16.9 s	52 - 54 tokens/s
45,398 tokens	67 - 68 s	38 - 41 tokens/s
The same 45K conversation, next turn	0.4 - 0.5 s	34 - 45 tokens/s

The first message of a chat is the slow part: it waits on the SSD for the model's experts. Follow-ups reuse what was already read. A 262K context costs about 8% against 131K, because more of the attention's memory streams from RAM.

The machine

Graphics card	NVIDIA GeForce RTX 3060, 12 GB (driver 610.57, CUDA 13.3)
Processor, RAM	Intel Core i5-12400F (6 cores, 12 threads), 30 GB
Disk	500 GB NVMe (WD Blue SN5000) on a PCIe 3.0 link; the model files live on it
System	Rocky Linux 9.6, built with gcc-toolset-13
Engine	Strata 0.1.45 (upstream, merged 17 Oct 2026) plus this fork's changes
Model	Qwen3.8-Flash-Next, IQ3_S (about 3.5 bits per weight, 78 GB of files) by ISTA-DASLab
Context	262K tokens (the longest prompt served so far: 114K)

Answer quality

Strata replaces some of a token's experts with similar ones already on the card to stay fast. In a small test (50 maths questions, 30 coding questions, answers read by hand): 47 of 50 and 30 of 30 correct, against 48 of 50 and 30 of 30 with nothing replaced. A test this small cannot show a difference of one or two answers either way. Since 24 Oct every JSON and code answer in the structured-output checks has been valid. Detail: SPEC.md.

MMLU-Pro (30 Oct): 87.1% - 183 of 210 questions (15 from each of its 14 subjects), thinking on, 32K context, temperature 0, the stand-ins on; every answer finished cleanly (no cut-offs or loops). For comparison, published Vals scores: Qwen3.8-27B 84.3%, Qwen3.8 Max 88.6%.

What you see

_{The Monitor: speed, card load and memory, the experts on the card, each request's hit rate.}

_{The Chat: under every answer, the number of tokens and the speed.}

How it works

The model has 512 experts per layer and picks 10 for every token; only some fit on the card. Everything this fork adds is about the ones that do not.

Stand-in experts. A missing expert is replaced by one already on the card only when router weight × logit gap ≤ budget. The budget tunes itself by measured speed, a token a stand-in could have flipped is recomputed exactly, and each request brakes itself when it sees flips.	A look-ahead that learns. Experts for 1, 2 and 4 layers ahead are guessed on the card and read from the SSD before they are needed. Each guess is the router's own score plus a small correction learned live from the real routing; its size is chosen per layer by a live contest, and it falls back to the plain router if it ever does worse. For Qwen3.8 IQ3_S setup downloads a trained one (release, 4 MB: 91 / 84 / 77% of the needed experts guessed 1 / 2 / 4 layers ahead) and it keeps learning on your PC; for other models it is off until STRATA_LA_LEARN=1 starts one.
Three tiers: card → RAM → SSD. The hottest experts on the card, the next in a RAM tier sized from free memory, the rest on the SSD. Swap rounds move a chat's experts onto the card in the background.	Long prompts, long memory. Prompts are read layer by layer, each layer's experts once, straight from the SSD. Past 64K the attention's memory streams from RAM, and a long conversation resumes from a saved state in under a second.
Draft and check. A small draft head guesses the next words; the big model checks several at once.	Nothing hand-set. Budgets, reserves of RAM and video memory, and what the look-ahead learned are measured on your PC and kept across restarts.

Needed experts guessed (live, 24 Oct)	Plain router	Learned look-ahead
1 layer ahead	77-79%	85-88%
2 layers ahead	66-69%	76-79%
4 layers ahead	55-58%	66-71%

Every change and its measurement: docs/HOTROD-DETAILS.md · the predictor study: docs/PREDICTOR-EXPERIMENTS.md · the full log of ideas tried: docs/perf-ideas.md · earlier experiments from a separate fork: docs/EARLIER-EXPERIMENTS.md.

New in v0.1.45-hotrod.2 (31 Oct): the learned look-ahead v2 (trained with a rank-weighted loss) is the one setup downloads; the expert placement comes from routing traces (tools/placement/solve.py); the Monitor's decode timeline runs only when you press Record (it cost 5-13% always on). Opt-in: the draft head's experts in slots with the rest computed exactly from RAM (STRATA_MTP_SLOTS, STRATA_MTP_ZC), and the look-ahead ring weighted by each layer's CPU cost (STRATA_RING_COST).

Every setting

Each model has its own config (strata-iq3_s.json here): engine settings go in its "env" (a blank value means the engine's default), engine flags in "args", server settings at the top level. Every switch, flag and server key, with its default and what it does: docs/FEATURES-qwen.md (generated from the code by tools/features_ref.py).

DeepSeek-V4.1-Flash too

The same ideas ported to upstream Strata's DeepSeek engine run DeepSeek-V4.1-Flash (Starw1's uncensored GSQ-RCO IQ2_XXS backbone with DeepSeek's MXFP8 Engram, 341 GB of files) on this same PC at about 5 tokens/s (4.7-5.5 on the calibration prompts; 4.8-5.0 without the ring and stand-ins). It lives on the upstream-dsv41 branch (upstream Strata 0.2.7.1 plus this fork's DeepSeek work), with its own engine, strata-dsv41; the two branches are kept apart and ported by hand.

What it adds to upstream's DeepSeek engine, on by default when the ring is on (=0 turns each off):

• A VRAM ring (DS4_RING=32): the learned look-ahead's guesses for the next 1-2 layers are copied from a page-locked RAM store into card slots; a pick uses its slot only if the copy has landed in time.
• Sticky ring (DS4_RING_STICKY): a landed copy stays until its slot is needed, so a later token that picks the same expert reuses it (about 80,000 reuses in one answer).
• The prompt pass's buffers as ring slots (DS4_LEND, DS4_LEND_RING): the 1 GB the chunked prompt pass keeps on the card holds 99-100 more ring slots while the model answers, and is handed back before a long prompt.
• A self-adjusting copy limit (DS4_RING_INFLIGHT, auto): copies in flight follow the measured decode speed (128-token averages, a 3% dead band; it settles at 11-12 on this PC). =fixed keeps 12.
• Stand-ins with their re-check and self-tuning budget, the learned look-ahead and the learned VRAM reserve, as on Qwen.
• The RAM store gives back its coldest idle chunk (unpinned first) when RAM runs short, and takes chunks back later.
• Pictures: with Unsloth's mmproj-BF16.gguf in the pack (--mmproj to pack_gguf.py) and --vision on, the engine's own vision tower reads images (319 MiB of VRAM; checked on a known picture: shapes, colours, positions and text right). Known issue: with the tower on, the RAM store is sized 0 and the ring turns off (a free-RAM drop is taken for a unified-memory machine); the fix is pending.

Measured and left opt-in: STRATA_LA_TAIL_READ=3 (a learned read-ahead of lower-ranked guesses: 23% slower, the drive is already at its limit here) and DS4_PCIE_BALANCE=1 (Qwen's dynamic balance ported: its copies arrive too late and it is not yet safe - do not turn it on). DeepSeek's limit on this PC is the NVMe at about 2.8 GB/s.

Install

You need a recent NVIDIA driver, the CUDA 13 toolkit, gcc-toolset-13 (Rocky Linux 9; other distributions are untested) and about 80 GB free on an NVMe.

sudo dnf install -y git gcc-toolset-13
git clone https://github.com/sangharshadhyeta/Strata-HotRod.git
cd Strata-HotRod
scl enable gcc-toolset-13 -- ./setup.sh --model IQ3_S --context 131072 --build --yes --no-start

• --build compiles this fork's engine; without it setup downloads upstream's, which lacks these changes. The compiled engine was checked against the one measured here: the same speed.
• For Qwen3.8 IQ3_S it also downloads the learned look-ahead's trained state (4 MB, SHA-256 checked) into data/.
• For 262K context set --max-context 262144 in the config's args.
• --model IQ3_S is the size tested here. Setup says it wants a 64 GB PC; naming it is the consent, and it runs on 30 GB.
• --models-dir /path/on/nvme puts the model files elsewhere.

Run inside a memory limit (94% of RAM; two uncapped runs once froze this PC):

systemd-run --scope -p MemoryMax=$(awk '/MemTotal/ {printf "%dK", $2 * 0.94}' /proc/meminfo) -p MemorySwapMax=0 \
  .venv/bin/python serve/server.py --engine strata --config strata-iq3_s.json \
  --port 8095 --host 0.0.0.0 --api-key YOUR_KEY

When it prints ready: http://127.0.0.1:8095/v1, open http://127.0.0.1:8095. For this PC only, leave out --host and --api-key.

Update: scl enable gcc-toolset-13 -- ./update.sh --build (new code, engine rebuilt where it changed, model kept).

DeepSeek-V4.1-Flash

About 345 GB free on an NVMe (139 GB of model, the 203 GB Engram table, 1 GB for pictures), the same toolchain, and the Qwen checkout above (the DeepSeek engine builds against its third_party/llama.cpp). These are the commands this PC was set up with; the branch's own setup.py --model Starw1-2.00bpw-MXFP8 downloads the same files and makes the pack, but that route is untested here.

# the DeepSeek branch beside the Qwen checkout (run from inside Strata-HotRod, where the Qwen install left you)
git fetch origin upstream-dsv41 && git worktree add ../Strata-HotRod-ds upstream-dsv41
cd ../Strata-HotRod-ds
PY=../Strata-HotRod/.venv/bin/python   # the Qwen install's venv runs this branch's server and tools too

# the engine (sm_86 = RTX 30-series; set yours)
scl enable gcc-toolset-13 -- bash -c 'cmake -G Ninja -S . -B build-dsv41 -DCMAKE_BUILD_TYPE=Release \
  -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=OFF -DSTRATA_BUILD_DSV41=ON -DSTRATA_NATIVE_EXPERTS=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86-real -DSTRATA_GGML_DIR=$PWD/../Strata-HotRod/third_party/llama.cpp \
  && ninja -C build-dsv41 strata-dsv41'

# the model (M: the folder for the files): Starw1's 4 shards + the MXFP8 Engram, Unsloth's metadata shard,
# and for pictures Unsloth's mmproj-BF16.gguf in $M/..
M=/mnt/nvme/models/deepseek41-flash/GSQ-RCO-IQ2_XXS
$PY - "$M" <<'EOF'
import sys
from huggingface_hub import hf_hub_download, snapshot_download
m = sys.argv[1]
snapshot_download("Starw1/DeepSeek-V4.1-Flash-1017-Uncensored-GSQ-RCO-IQ2_XXS-MXFP8-Engram-GGUF", local_dir=m)
hf_hub_download("unsloth/DeepSeek-V4.1-Flash-1017-GGUF", "DeepSeek-V4.1-Flash-1017-UD-Q8_K_XL-00001-of-00009.gguf",
                local_dir=m + "/unsloth")
hf_hub_download("unsloth/DeepSeek-V4.1-Flash-1017-GGUF", "mmproj-BF16.gguf", local_dir=m + "/..")
EOF

# the pack (a few MB of index beside the files; no weights are copied)
$PY tools/dsv41/pack_gguf.py --model $M \
  --shards DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-0000{1,2,3,4}-of-00004.gguf \
  --engram $M/DeepSeek-V4.1-Flash-Engram-MXFP8.gguf \
  --engram-meta $M/unsloth/DeepSeek-V4.1-Flash-1017-UD-Q8_K_XL-00001-of-00009.gguf \
  --mmproj $M/../mmproj-BF16.gguf --out /mnt/nvme/strata-data/packs/ds41v
cp data/dsv41/profile-2.00.bin /mnt/nvme/strata-data/packs/ds41v/expert-profile.bin

The config strata-ds41.json (the ring and stand-ins are switched on here; everything above follows from them):

{
 "exe": "/path/to/Strata-HotRod-ds/build-dsv41/strata-dsv41",
 "args": ["--pack", "/mnt/nvme/strata-data/packs/ds41v", "--max-context", "262144", "--eos", "1",
          "--vision", "on", "--dspark", "auto"],
 "cwd": "/path/to/Strata-HotRod-ds",
 "tokenizer": "/mnt/nvme/strata-data/packs/ds41v/tokenizer",
 "model_name": "deepseek-v4.1-flash",
 "arch": "deepseek41",
 "engine_ready_s": 3600,
 "env": {"DS4_RING": "32", "DS4_RING_PIN": "1", "STRATA_STANDIN": "1"},
 "answer_room": false,
 "reasoning_budget_tokens": 0
}

• "answer_room": false and "reasoning_budget_tokens": 0: no limit but the context (otherwise DeepSeek's thinking stops at 32,768 tokens and a client's cap keeps a quarter for the answer).
• --vision off (or leaving out --mmproj when packing) gives text only, and today the faster setup (see the known issue).
• Run it as Qwen, in the same memory limit, from Strata-HotRod-ds with $PY serve/server.py --engine strata --config strata-ds41.json. The first start at a new context size learns its VRAM reserve; if it prints ring: off (... would not fit), start it once more.

Using it

Any OpenAI-compatible client works with http://127.0.0.1:8095/v1 and your API key:

curl http://127.0.0.1:8095/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"model":"strata","temperature":1.0,"top_p":0.95,"max_tokens":300,
       "messages":[{"role":"user","content":"Name the three largest cities in India."}]}'

• Thinking is on by default; the chat menu picks off, low, medium or high.
• Sampling: with no temperature set, decoding is greedy. The model card recommends temperature 1.0, top-p 0.95, top-k 20 with thinking on, and 0.7 / 0.8 / 20 with it off.
• Reading while you type: the chat page sends your draft as you pause, so Send only reads what changed. Programs can do the same with "strata_prefill": true (or X-Strata-Prefill: 1, or /v1/prefill).
• Pictures (optional): add a "vision" section to the config; a message can then carry an image_url (docs/IMAGES.md).
• One request at a time; others wait their turn.

Every option: docs/DETAILS.md (upstream's reference; it covers more than we tested).

Not tested

• Any other graphics card, any AMD card, Windows, macOS or another Linux.
• Any other model or size (Q2_0, IQ2_XS, IQ3_XXS, Coder, Swift, Unsloth's files).
• Several requests at once, or two graphics cards.
• A prompt that fills the whole 262K window (the longest served: 114K tokens).

Credits and license

The model is Qwen3.8-Flash-Next by the Qwen team, compressed by ISTA-DASLab. Strata is by Niko1221 and its contributors; this fork keeps its MIT License and copyright notice. The engine uses parts of llama.cpp / ggml, and the picture encoder uses llama.cpp's mtmd. A few parts and every model have their own licenses: docs/HOW_IT_WORKS.md. Security: SECURITY.md.

Ideas this fork builds on

Several changes were first built and measured in solid.cpp, a llama.cpp fork for MoE expert caching, and ported here one concept at a time. Prior art behind them:

• leloch/llama.cpp (moe-cache-pr): the original GPU-resident MoE expert cache.
• thecodacus/llama.cpp (prefetch-experts, from the Codacus channel): expert prefetch overlapped with compute.
• JustVugg/colibri: LFRU eviction and the Expert Atlas method; its router-prediction experiment showed where a predictor stops paying.
• FlashML-org/FreeToken: the hit/miss split for expert uploads.
• antirez/ds4: the three-tier layout and explicit reads instead of mmap.
• aswinkumar99/qwen3.8-flash-next-expert-traces (CC-BY-4.0): the expert-predictor study on this model; the learned look-ahead uses its training target (the router's top-32 scores plus a hinge), and --expert-predictor on downloads its published predictor.
• vLLM: a reference for the model's code, and not reserving memory for unused context.
• Papers: FlexGen (2303.06865) and MoE-Lightning (2411.11217) for layer-by-layer prompts; Stream2LLM for reading the prompt while it is written; the expert-prefetch literature (Speculating Experts 2603.19289, PILOT / Fate 2502.12224, FineMoE, SP-MoE 2510.10302, ProMoE, PowerInfer-2).