Fault-Resilient-MPS

Companion artifact for the paper "Fault-Resilient MPS: Isolation and Fast Recovery Guided by GPU Fault Characterization" (anonymous submission). This repository contains the source code, fault-injection harness, and evaluation scripts for the two software mechanisms described in §5 and §6:

1. Memory fault isolation — a modified NVIDIA open kernel driver (nvidia-uvm.ko + nvidia.ko, v580.95.05) that intercepts fatal memory faults in the UVM page-fault handler, redirects the faulting virtual address to a dummy page so the GPU resolves the fault normally, and terminates only the faulty MPS client via the MPS daemon's native terminate client interface.
2. Fast recovery — an active/standby architecture for vLLM inference that combines vLLM sleep mode with the CUDA VMM API to share model weights and KV-cache physical pages across two processes, so the standby resumes from the active's current decode step without reloading model weights or replaying prefill.

Layout

fault-injection-module/        §5 + §6 fault triggers (build with `make`)
MMU-Fault-Isolated-Driver/     Modified NVIDIA open kernel driver
Fast-Recovery/                 §6 active/standby implementation (vLLM)
evaluations/                   Reproduction scripts for the headline numbers
  isolation/run_m1.sh            §8.1 M1 — 9 attackers × 2 isolation states
  recovery/run_exp1.sh           §8.3 Exp 1 — recovery time vs cold / sleep

The remainder of this README covers three tasks: (1) testing the fault-injection module, (2) installing the isolated driver, and (3) using the fast-recovery implementation.

---

1. Test the fault-injection module

cd fault-injection-module
make                                  # builds all attackers + victim
./test_all_interference.sh            # MPS sweep: each attacker vs. the victim

The harness starts the MPS daemon, launches the victim, runs the attacker, and reports ALIVE / DIED for the victim. With the stock driver every attacker takes down the victim; with the isolated driver below, memory-fault attackers no longer kill the victim. See fault-injection-module/README.md for the full list of fault types and the SM86 verification notes.

2. Install the isolated driver

cd MMU-Fault-Isolated-Driver

# Build all kernel modules
make modules -j$(nproc)

# Install (requires root) and reload
sudo make modules_install -j$(nproc)
sudo depmod -a

# One-shot rebuild + reload cycle (stops GPU services, unloads/reloads modules)
bash reinstall_modified_driver.sh

After reload, the isolation behaviour is gated by a module parameter so that the same kernel can be A/B tested:

# Enable fault isolation:
echo 1 | sudo tee /sys/module/nvidia_uvm/parameters/fault_isolation_on
# Disable (default — vanilla UVM behaviour):
echo 0 | sudo tee /sys/module/nvidia_uvm/parameters/fault_isolation_on

Reproduce the §8.1 isolation matrix:

cd evaluations/isolation
bash run_m1.sh 0       # baseline run, fault_isolation_on=0
bash run_m1.sh 1       # ours run,     fault_isolation_on=1
cat m1_iso0.tsv m1_iso1.tsv

Prereqs: modules loaded, uvm_enable_builtin_tests=1, sudo NOPASSWD, and the fault-injection binaries built under fault-injection-module/.

3. Use the fast-recovery implementation

cd Fast-Recovery/impl

# (one-time) build the libcuda interception layer
make -C libcuda buildtime

# Launch the active + standby pair for one model
PHASE6_MODEL="Qwen/Qwen3-0.6B" bash run.sh

run.sh starts active_server.py (the live vLLM inference server) and standby_proxy.py (a warm standby that imports the active's weight and KV-cache physical pages via the CUDA VMM API and enters vLLM sleep mode). Requests are routed through the proxy. If the active dies (e.g. an MPS client triggers a fatal RM fault and the shared CUDA context is destroyed), the proxy detects the failure, wakes the standby, and resumes the in-flight batch from the forward state the active had published — no model reload, no prefill replay. See Fast-Recovery/impl/README.md for the full architecture and environment variables.

Reproduce the §8.3 Exp 1 recovery-time table (5 models × {cold, sleep, ours}):

cd evaluations/recovery
MODELS_DIR=/path/to/local/Qwen2.5/models bash run_exp1.sh

Per-cell results land in exp1/<MODEL>/{cold,sleep,ours}/result.json.