T-060 · Local-inference acceleration: speculative decoding + single-GPU engine arbitration

[darthzen]

Model: sonnet (research/design + benchmark harness; live GPU-swap steps human-gated)
Creds: sa-nemoclaw-observe (metrics) · node GPU ops = human
Depends on: T-012 (agent running on qwen3-coder:30b)
Priority: optimization — not on the correctness path

Goal

Accelerate the local grunt-work layer (the point of the MCP interface: frontier models reason, local models do the busy work) without quality loss, via speculative decoding — and solve the single-V100 contention it creates.

Target model: qwen3-coder:30b (also serves Hermes + ollama-code-mcp — shared benefit).
Baseline: Ollama 0.30.6 on Tesla V100 32GB (supports the DRAFT Modelfile directive); no drafter < 4B currently pulled.

Path A — Ollama + drafter (drop-in, recommended first)

• Pull/create a small same-family dense drafter (e.g. qwen3:1.7b/0.6b), pair with qwen3-coder:30b via Modelfile DRAFT (+ --quantize-draft). OpenAI endpoint unchanged → zero agent-side change.
• Stretch: DeepSpec-trained drafter. DeepSeek's DSpark (2026-06-27) is DeepSeek-V4-only, but the co-released DeepSpec toolkit is MIT-licensed and trains drafters for Qwen3/Gemma — a purpose-trained drafter should beat a generic small model's acceptance rate. Evaluate.
• Benchmark on the V100: tokens/sec and draft acceptance rate vs no-draft baseline, on representative grunt-work prompts (tool-call loops, code edits). Expect 1.5–3× if acceptance holds. Confirm the CUDA runner (not just Mac/MLX) actually engages spec-decode at 0.30.6.

Path B — vLLM (higher ceiling, only if A underdelivers)

• Evaluate 1Cat-vLLM (Tesla V100 / SM70 fork: AWQ 4-bit + "Qwen3.6-class MTP speculative decoding" opt-in, validated on Qwen 27B/35B). This is the route to DeepSeek-style MTP on Volta.
• Cost: AWQ-quantize the model, separate vLLM deploy, fork-maintenance risk. NemoClaw has a managed-vLLM provider, so not exotic. V100 = Volta: no FP8, no fused EAGLE-3 kernels — draft/AWQ paths only; gains real but below Hopper headline figures.

The hard requirement — single-V100 engine arbitration (quiesce one while the other runs)

Ollama and vLLM cannot both hold the V100 (two ~30B models won't co-reside). Design mutually-exclusive GPU ownership with a clean handoff:

• Scheduler-enforced exclusivity: both engines request nvidia.com/gpu: 1; with one allocatable V100 the scheduler runs only one at a time (the other stays Pending until the GPU frees). Native mutual exclusion.
• Arbiter: scale the inactive engine to zero → verify VRAM actually drained (DCGM / nvidia-smi) → then scale the target engine up. Expose a switch action + demand-based trigger (workload type or explicit request).
• Fast-release levers: Ollama OLLAMA_KEEP_ALIVE=0 unloads models from VRAM on idle without pod churn; vLLM holds VRAM for its lifetime, so it must scale-to-zero (or use vLLM sleep mode if the fork supports it) to release.
• Not available on V100: MIG (Ampere+ only); MPS can't fit two 30B models. So it's exclusivity + fast swap, not co-tenancy.

Acceptance criteria

• Path A benchmarked on the V100: TPS + acceptance-rate table vs baseline; lossless output confirmed.
• A demonstrated clean engine handoff: Ollama⇄vLLM switch with VRAM verified free before start — no OOM, no orphaned GPU memory. Arbiter manifests/script committed.
• Go/no-go recommendation on Path B with numbers.
• Short writeup (GPU arbitration on a constrained single-accelerator node) — portfolio material.

Notes

• Shared-resource caution: the V100 also serves Open WebUI / niffler / ollama-code-mcp — live swaps are human-gated.