(Tutorial) - Serve recipes for 5090

[CommanderTurtle]

Have a 5090 and ultraspecced gaming rig? Thank me later guys, as I've been modeling and exploring variants for a while now.

Just edit serve command:

• This one is - compute1/Agents-A1-GPTQ-INT4-Sym

CUDA_VISIBLE_DEVICES=0
vllm serve compute1/Agents-A1-GPTQ-INT4-Sym
--host 0.0.0.0
--port 8000
--tensor-parallel-size 1
--max-model-len 131072
--gpu-memory-utilization 0.86
--kv-cache-dtype fp8
--max-num-seqs 2
--language-model-only
--tool-call-parser qwen3_coder
--reasoning-parser qwen3
--enable-auto-tool-choice
--trust-remote-code

Also, not as-perfect in results, but still qwen-based and godly. Some abliterated-s had been made and worth researching if needed: ⬇️

• The second one is - groxaxo/Qwen-AgentWorld-35B-A3B-GPTQ-Pro-Int4

CUDA_VISIBLE_DEVICES=0
vllm serve groxaxo/Qwen-AgentWorld-35B-A3B-GPTQ-Pro-Int4
--host 0.0.0.0
--port 8000
--tensor-parallel-size 1
--max-model-len 131072
--gpu-memory-utilization 0.73
--dtype auto
--max-num-seqs 2
--language-model-only
--tool-call-parser qwen3_coder
--reasoning-parser qwen3
--enable-auto-tool-choice
--trust-remote-code

Specs: 64gb ddr5, and 5090.

---

get the cude on wsl? (generic debian-fork instructions):

• Cuda install made easy, not reliant on .debs..

sudo apt update
sudo mkdir -p /etc/apt/keyrings
# Correct Add-Cuda-Repo Method:
deb [signed-by=/etc/apt/keyrings/cuda-wsl.gpg] https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/ /
# Verify your keyring file exists
ls -l /etc/apt/keyrings/cuda-wsl.gpg
# If it’s missing, recreate it..
curl -fsSL https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/3bf863cc.pub \
| sudo gpg --dearmor -o /etc/apt/keyrings/cuda-wsl.gpg
# Fix the repo file cleanly
sudo nano /etc/apt/sources.list.d/cuda-wsl.list
# Replace the contents with:
deb [signed-by=/etc/apt/keyrings/cuda-wsl.gpg] https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/ /
# Save → exit → then run:
sudo apt update
# This installs the **WSL‑optimized CUDA toolkit**, not the generic Linux one:
sudo apt install -y cuda
# SUDO TIMEOUT:
-Defaults       timestamp_timeout=0
# Policy against SHA1 warning?
# https://neilzone.co.uk/2026/01/dealing-with-apts-warning-policy-will-reject-signature-within-a-year-see---audit-for-details/
# "The “correct” way of solving this is for the repository provider to update their signing key to something which is compliant. 
# However, I have no control over what a repository provider does, or when they will do it. For instance, the warning message above 
# suggests to me that I will have a problem on 1 February 2026, so under a month away. I can suppress this warning - and tell apt to accept 
# the key - by adding -o APT::Key::GPGVCommand=1 to apt update: apt update -o APT::Key::GPGVCommand=1 Or, to avoid having to add 
# that each time, I can added a slightly-tweaked version of it to an apt config file, in /etc/apt/apt.conf.d. For instance, I can put this into 
# /etc/apt/apt.conf.d/25keypolicy: APT::Key::GPGVCommand "1"; Hopefully though, repository providers will update their keys (which will 
# then need re-importing)."
sudo nano /etc/apt/apt.conf.d/25keypolicy
# Add line, save and exit:
APT::Key::GPGVCommand "1";

nvidia-smi

Thank me later bois.