Skip to content

SETUP.md

Latest commit

 

History

History
252 lines (216 loc) · 23.4 KB

SETUP.md

File metadata and controls

252 lines (216 loc) · 23.4 KB
Raw

Setting up Maya

The setup in detail: what its first run does, starting again and updating, its options, and what to do if the build fails. Back to the README.

What the first run does

It first asks which model to set up:

  • GLM-5.3 (744B) - Maya's own Maya-XS GGUF (--model glm-5.3-maya-xs) or unsloth's UD-IQ2_M GGUF (--model GLM-5.3-UD-IQ2_M), on the same engine; its page;

  • GLM-5.3-Flash - Project Maya's GGUF quants (or turboderp's EXL3 quants), from one GPU up; the steps below;

  • DeepSeek-V4.1-Flash-1017 - LibertAIDAI's EXL3 quant at 2.9 bits (211 GB) or at 3.5 bits (258 GB, about 280 GB of VRAM and RAM; --ds4-quant exl3-3.5, its own folder; a second quant hard-links the Engram tables from the first instead of downloading them) and its two Engram tables from DeepSeek's repo (203 GB), on its own engine (build/maya-ds4): Linux, NVIDIA V100 or newer (pictures: RTX 30 / A100 or newer), and AMD. It is fast with about 230 GB of VRAM and RAM together; with less, the setup warns and the engine reads the experts that do not fit from the model's files on the SSD (--mmap-experts auto), much slower - one RTX 3090 with 64 GB of RAM: prompts about 50 tokens/s, answers 4-5 tokens/s (faster with --dspark none). The same steps with these differences: the contexts are 32K-1M (256K recommended where the PC's VRAM + RAM hold the model's files, else 64K; 512K and 1M take VRAM from the expert cache, and a prompt that long reads slowly - the indexer's work grows with the square of its length); step 5 shows both sources and checks every file against its pinned size (and, once downloaded, its sha256: tools/ds4/download.json, download-3.5bpw.json); --model-dir DIR uses files you already have; the pack is DIR/maya-ds4/ (tools/ds4/pack.py, about a minute); pictures need nothing more - the model's own vision tower runs in the engine; the config is maya-ds4.json, with tensor parallel across pairs of equal GPUs (--tp 2). The engine balances its own layer split; ./maya.sh --calibrate (not offered by the setup) measures each --tp that groups your GPUs differently and fewer CPU threads, an engine start each (about an hour on 9 GPUs), and keeps the fastest answers whose long prompts still read at 60% of the config's own --tp's speed; a setting the engine does not start with is left out (tools/calibrate_ds4_tp.py; the result goes into maya-ds4.json and a setup again keeps it).

    Or from unsloth's GGUFs (--ds4-quant; the setup asks): UD-Q8_K_XL (508 GB, the default: dense weights BF16, routed experts MXFP4, the Engram tables FP8 - DeepSeek's own formats); its UD-Q4_K_XL (501 GB: dense weights Q8_0, less VRAM) is no longer in the setup's list - --ds4-quant gguf-q4 and an install of it work as before. The same engine reads them in place (the pack is tools/ds4/pack_gguf.py's), from NVIDIA Volta (V100) on: the most-used experts in VRAM, the others read from the GGUF through the OS page cache - from RAM where they fit, from the SSD otherwise - so it runs whatever the memory (more VRAM and RAM is faster; keep the files on an NVMe SSD; --mmap-experts off copies as many of them into RAM as fit instead, on huge pages split across the CPU sockets' memory - measured no faster than the page cache). Prompts go through in chunks of 8192 tokens, also where tensor parallel falls back to one GPU per layer group (the log says tensor parallel 2 -> 1, as with the Maya quants) - before v1.1.17 those ran every prompt in windows of 16 tokens, an order of magnitude slower on long prompts (a community measurement, 7x RTX 3090, Maya-S read in place: 32K tokens 99 tokens/s in windows, 1010 in chunks). The GGUFs have no DSpark drafter and no vision tower.

  • MiniMax-M3.1-Flash (--model MiniMax-M3.1-Flash) - Mia-AiLab's EXL3 quant at 3.5 bits (197 GB with MiniMax's DSpark drafter; every file pinned and checked, tools/mm3/download.json) on its own engine (build/maya-mm3): NVIDIA from V100 on, Linux and Windows, and AMD (as GLM's EXL3 quants); Maya's GGUF quants (--model minimax-maya-xs / minimax-maya-s: SETTINGS.md) run on both too. About 250 GB of VRAM and RAM hold all of it; with less the setup warns. The same steps with these differences: the contexts are 32K-1M, the recommended one the largest (up to 256K) whose KV cache fits beside the model in this PC's VRAM + RAM, its RAM said, with warnings past 256K and when the cache would take the experts' RAM; the pack is DIR/maya-mm3/ (tools/mm3/pack.py, Maya's own copy of MiniMax's chat template inside it); pictures are on - the model's vision tower runs in the engine (--no-vision: off); the config is maya-mm3.json, with tensor parallel in groups of 4 or 2 where the GPUs make whole groups, KV streaming on, MiniMax's sampling (temperature 1.0, top-p 0.95) and thinking level Auto; then the DSpark drafter is measured on this PC (--drafting chooses without measuring), as DeepSeek's. Its settings: SETTINGS.md.

    Or orcarouter's abliterated model (DeepSeek-V4.1-Flash-1017 with its refusal direction taken out of the weights), in two versions: Mix-128G (--ds4-quant abliterated-128g; experts ~1.6 bits, ~114 GB of VRAM + RAM) and Mix-256G (--ds4-quant abliterated-256g; experts 2.5-4.25 bits by layer - IQ2_S / IQ3_XXS / IQ3_S / MXFP4 - ~236 GB of VRAM + RAM), in <models>/DeepSeek-V4.1-Flash-1017-abliterated-GGUF/<version>; --ds4-quant abliterated picks the one your PC holds. Their n-gram table (shard 2, not abliterated) is not downloaded: both read Maya's Q6_K tables (161 GB, in Engram-Q6_K beside them - linked from a Maya DeepSeek quant on the same drive, else downloaded once). Its repo is gated: accept its terms on Hugging Face, then set HF_TOKEN or run hf auth login - the download sends that token (curl reads it from a file only you can read). It is served as deepseek-v4.1-flash-abliterated, with DeepSeek's DSpark drafter and vision tower as for unsloth's GGUFs (the drafter only proposes tokens - the model checks each one).

GLM-5.3-Flash's first run takes 20-40 minutes plus the download:

  1. checks the PC (GPUs, driver, CUDA toolkit, compiler, RAM, CPU);
  2. asks which GPUs to use (all of them by default, up to 16), how much context (32K recommended), which model to download: GLM-5.3-Flash · Maya-S (recommended; Maya-S24 on cards of 24 GB or less), Maya-M, Maya-M-Derisked or Maya-L, and how it drafts: the model's MTP block (the default) or a DFlash2 drafter (NVIDIA RTX 30 / A100 and newer, and one AMD Ryzen AI Max APU);
  3. installs its Python packages into .venv and gets llama.cpp's source at a pinned commit (it lists both and asks);
  4. compiles the engine for your GPU(s) (10-30 minutes, once);
  5. the model: it shows the source, the size (Maya-S: 96.5 GB) and the exact curl commands, and downloads only when you answer y; every file is checked against its published sha256. You can run the commands yourself instead, or use files you already have: ./maya.sh --gguf-dir DIR - other quants than these four are experimental: they run, but Maya is not measured with them. With DFlash2 it also downloads the drafter (0.7 GB, checked against its sha256) next to the models;
  6. builds the pack - the engine's index of the model files, about 1 GB, written into the model folder. A Maya model's files that were already there are checked against their sha256 first (once each); the pack records the files, and a start refuses it if they change afterwards (--repack builds it again);
  7. pictures: compiles the image encoder (10-20 minutes, once) and fetches its files (1.1 GB, shown and asked first); --no-vision skips it;
  8. writes maya-<model>.json and run-maya-<model>.sh; with DFlash2 it runs the speed calibration (5-20 minutes: DFlash2 stays only if it is faster on your PC); then the settings tuned for this PC earlier, or it offers the tuning (about 10-15 minutes; ./maya.sh --calibrate any time) - and starts the dashboard.

The start takes a few minutes: the engine pins most of the free RAM (all but about 6 GB) for its expert tier and warms its caches. Other programs get little RAM while Maya runs.

Linux: systemd-oomd. Stock Ubuntu 22.04 and newer run systemd-oomd, which ends a desktop session's whole terminal (Maya, and the setup's own tuning with it) when the session's memory pressure stays over 50% for 20 seconds. Loading a large RAM tier - about 200 GB of DeepSeek experts - reclaims the page cache for minutes and can cross that, with RAM still free and nothing in dmesg; only the journal says systemd-oomd killed ... process(es) (issue #497). --check and the setup warn when it runs. Stop it for the run with sudo systemctl stop systemd-oomd (it starts again at the next boot), or exempt your user sessions for good with a drop-in, then log out and in:

sudo mkdir -p /etc/systemd/system/user@.service.d
printf '[Service]\nManagedOOMMemoryPressure=auto\n' | sudo tee /etc/systemd/system/user@.service.d/90-maya-oomd.conf
sudo systemctl daemon-reload

RDNA4 EXL3 prefill

On an R9700 / RX 9070-family GPU, grouped EXL3 prompt processing can use the RDNA4 matrix kernels, with 16-row tiles (opt-in for now: EXL3_WMMA_GFX12=1). Guided setup shows how to save the settings through --env or an existing configuration, verify the build and tests, and switch back for comparison. This applies to EXL3 expert weights, not GGUF, and floating-point results can differ from the portable path.

Next time, and updating

Next time, just run ./setup.sh again (or the model's ./run-maya-<model>.sh): it starts right away, nothing is downloaded twice, on the same screen - its dashboard shows the Monitor's numbers (the speed, where the experts live, the GPUs, the requests), Tab its log. Ctrl+C stops it - and so does any other way of ending maya.py: kill, a service manager, closing the terminal or the window, even a crash: the server and the engine end with it (the engine keeps its conversation where there is time for that).

Updating: the dashboard's About says when a new version is out, and its Update button does it: Maya downloads the new code (git), compiles only the engine files that changed and loads the same model again - the model is not downloaded again, and your settings stay. By hand: git pull, then ./setup.sh. (The check asks GitHub for the latest release, at most every six hours; MAYA_UPDATE_CHECK=0 turns it off.)

Running Maya as a systemd service

Set the model up and start it by hand once first, then let a unit start it at boot. The unit runs the config the setup wrote (maya-<model>.json, in the Maya folder):

[Unit]
Description=Maya
After=network-online.target
Wants=network-online.target

[Service]
User=you
WorkingDirectory=/path/to/maya
ExecStart=/path/to/maya/maya.sh --config /path/to/maya/maya-<model>.json
Restart=on-failure
RestartSec=30
TimeoutStartSec=3600
TimeoutStopSec=600

[Install]
WantedBy=multi-user.target
  • Time to start and stop. A big model at a long context takes a long time to load (the engine pins its expert tier in RAM and warms its caches), and the more context you give it (1M tokens, say), the longer. systemd ends a unit that is not up inside TimeoutStartSec and starts it again, which then loads again, for ever. Set it well above the slowest start you have seen by hand (TimeoutStartSec=infinity turns it off). TimeoutStopSec is the time Maya has to save its conversations and free the GPUs after the stop signal; give a big model several minutes.
  • Run the calibration once, by hand, first. A start measures nothing when nobody is there to wait (a service, a pipe, --yes): it prints one line naming what it skipped and starts with the engine's defaults. Run the calibrations yourself in a terminal while the service is stopped (./maya.sh --config maya-<model>.json --speed-calibration --no-start, and for MiniMax --calibrate --no-start too). They load the model a few times and keep the answer in the config, so the service starts straight from it afterwards. A terminal start of the same config still measures what is missing, as before.
  • Logs. The server's output (the same lines a terminal shows) goes to the journal: journalctl -u maya -f. The engine writes its own log to the file named by "log" in the config, maya-<model>.log beside it by default.
  • The model needs the whole machine: stop the service before a calibration, a setup or --check, and do not start two units on the same GPUs.

Options

Option
--setup set up again (other GPUs, context, model folder)
--check only check the PC
--gguf-dir DIR use GLM-5.3-Flash GGUF files you already have: their folder, or the .gguf file (the first one of a split model) when the folder holds several models. The folder must be writable: the pack goes inside it. A GGUF Maya does not list (a pruned model such as RCO-152E included: the shipped expert profile is for 288 experts a layer and is left out of any other count) gets its own sampling (general.sampling.*) and chat template from the file, else GLM's
--models-dir DIR where downloaded models go, each in a folder named after it - DIR/Maya-L/ holds that model's GGUF files (default ../Maya-data/models); put it on the NVMe
--import-model FOLDER use model files you downloaded some other way (a download manager, a copy from another PC, a Hugging Face cache): finds Maya's published files in FOLDER and its subfolders by size and sha256 - their names do not matter - and moves them into --models-dir's layout (same drive) or copies them (another drive, a cache link: --copy; the originals stay). Prints what was found, what is missing and what did not match; files that did not match are never touched, and a file already in place is never replaced. Moved files are recorded as verified
--copy with --import-model: copy the files that cannot be moved
--download-model download the model without asking (the commands and size are still printed)
--model NAME which model and files, without asking. Maya's quants by their setup id - the model first, then the tier (XS, S, M, L by the experts' bits): glm-maya-s, glm-maya-s24, glm-maya-m, glm-maya-m-derisked, glm-maya-l, deepseek-maya-xs, deepseek-maya-s, deepseek-maya-m, minimax-maya-xs, minimax-maya-s (or the full name, e.g. MiniMax-M3.1-Flash-Maya-XS). Or a model, its files asked: DeepSeek-V4.1-Flash-1017, MiniMax-M3.1-Flash. The names used before still work: Maya-S-v2-IQ2_XXS (GLM-5.3-Flash · Maya-S), Maya-S24, Maya-M, Maya-M-Derisked, Maya-L, and EXL3-3.05bpw / EXL3-4.05bpw / EXL3-sc_4.10bpw - turboderp's EXL3 quant of GLM-5.3-Flash at 3.05 or 4.05 bits per weight, or the self-calibrated one at 4.10 with 8-bit attention and shared experts (3.05bpw, 125.3 GB; 4.05bpw, 165.2 GB; sc_4.10bpw, 167.8 GB; all with the MTP block) that the engine reads in place from its safetensors: NVIDIA from V100, and AMD; none measured against the FP8 model (EXL3-sc_4.10bpw is not in the setup's list: named by hand, or set up before, it works as before)
--model-dir DIR use DeepSeek-V4.1-Flash-1017's files you already have: the folder with a GGUF's shards, or with the EXL3 quant's files and the Engram shards model-00047-of-00048.safetensors / model-00048-of-00048.safetensors of DeepSeek's repo (it must be writable: the pack goes inside it). Or an EXL3 folder of GLM-5.3-Flash or MiniMax-M3.1-Flash at any bit rate that is not one of the listed downloads (turboderp's other branches, Mia-AiLab's other bit rates): the setup reads the folder's own config.json and model.safetensors.index.json, checks each file whole by its header, never offers the listed downloads for it, writes a config named after the folder (maya-<folder>.json) and uses the folder's own sampling (generation_config.json) and chat template (chat_template.jinja) - GLM's / MiniMax's where the folder has none. MiniMax: --model MiniMax-M3.1-Flash --model-dir DIR is the same
--ds4-quant Q DeepSeek-V4.1-Flash-1017's files: maya-s, maya-xs (Maya's quants, the same as --model deepseek-maya-s / deepseek-maya-xs; Maya-S the default), gguf-q8 (unsloth's UD-Q8_K_XL GGUF), gguf-q4 (its UD-Q4_K_XL: dense weights Q8_0; not in the setup's list, named by hand), exl3 (LibertAIDAI's EXL3 quant at 2.9 bits), exl3-3.5 (its 3.5-bit quant: closer to the original, about 280 GB of VRAM and RAM), abliterated-128g / abliterated-256g (orcarouter's abliterated model as its Mix-128G / Mix-256G GGUF, with Maya's Q6_K n-gram tables; abliterated: the one this PC holds; a gated repo: HF_TOKEN or hf auth login); the EXL3 quants need NVIDIA V100 or newer (pictures: RTX 30 / A100 or newer) and are offered only there, and on AMD (as GLM's EXL3 quants there; docs/EXL3_HIP.md)
--mm3-quant Q MiniMax-M3.1-Flash's files: exl3 (Mia-AiLab's EXL3 quant at 3.5 bits, NVIDIA), maya-xs or maya-s (Maya's GGUF quants, the same as --model minimax-maya-xs / minimax-maya-s; gguf-xs / gguf-s before)
--no-vision text only: no image encoder (DeepSeek, MiniMax: their vision tower off; Qwen-3.8-Flash-Next: no mmproj download)
--draft mtp / --draft dflash2 how the model drafts tokens ahead: its MTP block (the default), or a DFlash2 drafter - downloaded (0.7 GB) into DIR/GLM-5.3-Flash-DFlash2/ and passed as --dflash (Maya-L: +44% on nine GPUs, +12% on two, +19% on one GPU; NVIDIA RTX 30 / A100 and newer, and one AMD Ryzen AI Max APU (Strix / Gorgon Halo) - older cards and other AMD GPUs keep the MTP block unless you pass this; the drafter's license is CC BY-NC-ND 4.0, non-commercial use only). The speed calibration then keeps it only if it is faster on your PC
--gpu N / --gpus 0,1,2,3 one GPU, or several (up to 16) that split the model's layers
--context N context length in tokens: 8192, 32768 (default), 65536, 131072; DeepSeek: 32768, 65536, 131072, 262144, 524288, 1048576 (the default 262144 where VRAM + RAM hold the model, else 65536); MiniMax: the same, the default by this PC's memory
--vram-reserve-mib N VRAM in MiB the expert pool leaves free on each GPU (default 700), kept in the config - more where the desktop or other programs on the GPU need room (the expert pool's size)
--port N, --host 0.0.0.0 --api-key KEY another port; reachable from your network (always set a key; several: KEY1,KEY2)
--env KEY=VALUE an engine setting kept in the config (see Tuning)
--host-compiler g++-12 when your g++ is newer than your CUDA accepts ("unsupported GNU version")
--rebuild, --repack compile the engine / build the pack again
--calibrate tune the engine for this PC (see Tuning): GLM's CPU lane, DeepSeek's --tp and CPU threads, MiniMax's prefill calibration (the chunk and the matrix products' library a prompt is read with); then start; with --no-start only tune
--speed-calibration measure again whether the drafter (DFlash2, or DeepSeek's or MiniMax's DSpark) decodes faster on this PC (how; 5-30 minutes, with Maya stopped) and keep the faster, then start; with --no-start only measure
--drafting measure|on|off the drafter without the question: measure = the speed calibration (the default with --yes), on = draft with it, off = no drafting - both skip the measuring
--no-speed-calibration the same as --drafting on
--yes the recommended answers (the model download still needs --download-model)
--plain the setup in plain text with typed answers, and Maya in the terminal, not on their own screen (a pipe gets this too, and --yes a plain setup); the setup screen's whole output is in maya-setup.log

Web pages on another address

A browser stops a web page served from another address (a local chat client on http://127.0.0.1:8090, say) from calling Maya's API unless Maya allows that page. List the pages in the config, maya-<model>.json:

"cors_origins": ["http://127.0.0.1:8090"]

Exact addresses (scheme, host and port, no path); ["*"] lets every page in - set an api_key with it, or any page you open could use your model. It opens /v1/* and nothing else: the settings and the tools stay for Maya's own page. Without it nothing changes. A new --setup writes the config again: add the line back after it.

Giving the GPU back when idle

To share the PC with other GPU programs (another model server, games), let Maya unload the model after a while without requests. In the config, maya-<model>.json:

"idle_unload_s": 900,
"env": {"MAYA_GLM_SLOT_KEEP": "1"}

(--idle-unload 900 on the server's command line does the same.) After that many seconds with no request running or waiting, the engine ends: its GPU memory and its pinned RAM tier are given back (an RTX 3090 with Maya-S24: 22.9 GB of VRAM and 79 GB of RAM). The dashboard and /health say unloaded, and /v1/models still lists the model with the status unloaded. The next request loads the model again - about a minute before its first token - and the experts in VRAM warm up again over the next answers. With MAYA_GLM_SLOT_KEEP=1 the conversation in the model is written to the disk on the way out and read back, so the next turn of the same chat reads only its new tokens (1,630 of 1,690 prompt tokens reused after an unload). 0 (the default): never. A new --setup writes the config again: add the lines back after it.

If the build fails

  • "nvcc ... cannot build for these GPUs": Volta needs CUDA 12.x, Blackwell 12.8 or newer. The setup takes the newest installed toolkit that fits.

  • "unsupported GNU version": ./maya.sh --setup --host-compiler g++-12 (install g++-12 first).

  • cuda_profiler_api.h or nvtx3/nvToolsExt.h is missing: Maya's CUDA build needs the profiler API and NVTX headers as well as nvcc and the runtime. Install those development components for your toolkit. With a split CUDA 13 toolkit, NVIDIA's official header wheels can also stay in a project-local folder:

    python3 -m pip install --target .cuda-headers \
  nvidia-cuda-profiler-api==13.3.27 nvidia-nvtx==13.3.29
CPATH="$PWD/.cuda-headers/nvidia/cu13/include${CPATH:+:$CPATH}" ./maya.sh --setup

    Select wheel versions that match your toolkit. This include layout was checked with CUDA 13.3, nvidia-cuda-profiler-api==13.3.27 and nvidia-nvtx==13.3.29; other CUDA major versions can use a different layout. Use the same CPATH when rebuilding after an update. No shared toolkit files need changing.