H100 node - running models on multiple GPUs using anvailing

[K4TEL]

MiniMaxM3 FP8 444Bg - one to test

Figure out how to run it in a single node using multiple GPU cards available

[K4TEL]

Identifying the model

MiniMax-M3 is real and very recent (released June 1, 2026). The numbers check out against what you wrote: Total Parameters: 428B, Active Parameters: Approximately 22B per token (A22B), MoE/multimodal (vision-language). The quantized MXFP8 build is around 440GB — that's almost certainly what "FP8 444Bg" refers to (the FP8 checkpoint's on-disk/VRAM footprint in GB, not literally 444 billion parameters — the param count is ~428B). It's the model MiniMax shipped right after M2, with a new sparse-attention mechanism (MSA) that's the headline feature for the 1M-token context claim.

Memory math for an H100 node

First thing worth confirming before anything else: run nvidia-smi --query-gpu=name,memory.total --format=csv and nvidia-smi topo -m on the actual node to know whether it's 8×80GB SXM5 (NVLink/NVSwitch — the normal DGX/HGX H100 config) or fewer cards, or PCIe-only cards (much slower all-reduce for tensor parallelism). Everything below assumes the standard 8×80GB SXM5 topology, since that's what "H100 node" almost always means in practice.

With ~440GB of FP8 weights and 8×80GB = 640GB total VRAM: tensor-parallel across all 8 GPUs is the right shape, landing at roughly 55GB of weights per card and leaving ~25GB/card headroom for KV cache and activations. Runs on NVIDIA Hopper/Blackwell is confirmed on the model card itself, and the canonical recipe targets exactly this: vllm serve MiniMaxAI/MiniMax-M3 --tensor-parallel-size 8 --block-size 128 --tool-call-parser minimax_m3 --reasoning-parser minimax_m3 --enable-auto-tool-choice, or with expert parallelism added: vllm serve MiniMaxAI/MiniMax-M3 --tensor-parallel-size 8 --enable-expert-parallel.

Recommended setup

Framework: vLLM or SGLang, not Ollama. This is the key strategic point for the issue — Ollama's layer-split mechanism (what you used for the GLM-4v-9b/Qwen pipeline) doesn't do true multi-GPU tensor parallelism with collective NVLink ops; it's not built for sharding a single 428B model cooperatively across 8 cards. vLLM and SGLang are.

# vLLM — practical default for ATRIUM's likely context lengths (not the full 1M)
vllm serve MiniMaxAI/MiniMax-M3 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --block-size 128 \
  --kv-cache-dtype fp8 \
  --max-model-len 131072 \
  --tool-call-parser minimax_m3 \
  --reasoning-parser minimax_m3 \
  --enable-auto-tool-choice

A few specific reasons for each flag, tied to your hardware:

• --tensor-parallel-size 8: required to fit ~440GB of weights across 80GB cards with any KV-cache headroom at all.
• --enable-expert-parallel: for a MoE model, plain TP replicates the routing/non-expert weights on every GPU and shards everything identically; expert parallelism instead spreads the 256 experts across GPUs so you're not wasting VRAM on redundant copies — without it, the framework falls back to tensor parallelism, which replicates all expert weights on every GPU and wastes memory on non-active experts. With only ~25GB/card of headroom, this matters.
• --kv-cache-dtype fp8: adds roughly 1.5× to the KV pool — reported lossless across the full native context — cheap way to buy back context budget.
• --max-model-len 131072 instead of the full 1M: set max-model-len to the longest prompt + output you actually serve; a smaller window frees KV-pool headroom for higher concurrency and lets the model fit on fewer GPUs. For chunked archival-text enrichment (your TEATER/ArchaeoBERT prompts are short, dynamically-injected vocab subsets, not 1M-token documents), there's no reason to reserve memory for a context window you won't use — capping this is real, recoverable VRAM.

SGLang alternative, if you want to A/B it (some report better MoE throughput here):

python -m sglang.launch_server \
  --model-path MiniMaxAI/MiniMax-M3 \
  --tp 8 --enable-moe-ep \
  --quantization fp8 \
  --mem-fraction-static 0.85 \
  --context-length 131072 \
  --reasoning-parser minimax-m3 --tool-call-parser minimax-m3

Deployment risk to flag in the issue

Support is brand new — MiniMax-M3 support has not yet shipped in a stable vLLM release; use the dedicated Docker image or nightly. Before committing to this for any pipeline integration, pin and test against the exact vLLM/SGLang build that has confirmed MSA-attention support — don't assume pip install vllm on a stable release will work day one.

How this would slot into your existing architecture

vLLM and SGLang both expose an OpenAI-compatible HTTP endpoint, same shape as Ollama's API — so the llm_config.txt / OLLAMA_HOST hot-swap pattern your enrichment pipeline already uses could point at a vLLM endpoint with minimal client-side changes. The bigger open question for the issue is whether constrained/structured decoding (your LogitMatch + Pydantic JSON-schema requirement) carries over cleanly — vLLM supports guided/structured decoding via its own backend, but you'd want to verify schema-constrained generation actually works correctly with this specific MoE+MSA architecture before treating M3 as a drop-in upgrade over Qwen 14B for the TEATER/ArchaeoBERT extraction task, rather than assuming it on faith.