ElasticServe

ElasticServe is an experiment-first control plane for SLO-aware, multi-tenant LLM serving on a single NVIDIA DGX Spark. It sits in front of an OpenAI-compatible backend and studies how admission, queue ordering, and gateway-level concurrency affect latency SLO goodput and tenant fairness under overload.

The first milestone deliberately uses one model and one pinned vLLM engine. That keeps the comparison causal: FCFS and weighted-fair policies see the same backend, workload trace, and engine configuration. Multi-model residency, SGLang, and GPU partitioning are later experiments, not MVP dependencies.

Current status

M2 is running on the target DGX Spark:

• immutable request and tenant-policy contracts;
• deterministic FCFS and weighted-fair schedulers;
• seeded open-loop workload generation;
• SLO-goodput, percentile, and Jain-fairness summaries;
• a CPU-only discrete-event simulator for testing policies before using the GPU;
• a pinned DGX Spark FP8/vLLM baseline and saturation sweep;
• a streaming OpenAI-compatible FCFS gateway with bounded admission and metrics;
• a measured queue-wait admission deadline;
• a mixed-length pilot and calibrated prefill/decode relative-cost model.

Across five paired, previously unused seeds at 16 offered req/s, a 750 ms queue budget produced a median 6.79x goodput improvement over admit-all FCFS. Candidate p95 TTFT remained below the declared 1,000 ms SLO in every run; median rejection rate was 23.5%, with rejections included in all-offered accounting. See docs/validation.md for the exact workload, controls, bootstrap intervals, tradeoffs, and raw-artifact locations. This remains an experimental gateway, not a production release.

The same fixed deadline also improved one mixed 64–960 input / 32–224 output token pilot from 0.78 to 4.50 goodput req/s. A four-point single-concurrency profile now supplies separate input-prefill and output-decode cost coefficients for the next weighted-fair experiment.

M3's first paired study is complete: the gateway now supports QUEUE_POLICY=wfq (start-time-fair queueing weighted by tenant policy and the calibrated service-cost model), and elasticserve-loadgen replays seeded multi-tenant traces with exact token-id prompts and tenant metadata headers. Across five paired seeds at 8 req/s mixed-length overload behind the same 750 ms admission rule, weighted-fair ordering left aggregate goodput statistically unchanged while raising Jain fairness in every seed (paired median +0.15) and minimum-tenant attainment by a median 11 points: gold's rejection share fell to 0–15% while weight-1 batch absorbed the rejections. Ordering decides whose requests meet their SLO, not how many complete.

Quick start

The core has no third-party runtime dependency.

python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e .
python -m unittest discover -s tests -v

elasticserve simulate --policy fcfs --seed 7 --requests 300
elasticserve simulate --policy wfq --seed 7 --requests 300

Run both simulations with the same seed. Their workloads are identical; only dispatch order changes.

DGX Spark baseline

Use NVIDIA's current DGX Spark vLLM playbook to select a compatible image and model. Do not use a floating latest tag for recorded experiments.

export VLLM_IMAGE=nvcr.io/nvidia/vllm:<version-or-digest>
export MODEL=<hugging-face-model-handle>
export HF_TOKEN=<token-if-required>
./scripts/start_vllm.sh

For a background server, set DETACH=1. The detached container is intentionally not auto-removed so its logs remain available; stop and remove it explicitly with docker rm -f elasticserve-vllm after the run.

In another terminal:

curl -fsS http://localhost:8000/health
./scripts/capture_environment.sh results/raw/environment.txt

The engine's --max-num-seqs is a launch-time ceiling. ElasticServe controls admission and queue policy outside that ceiling; it does not claim to hot-reconfigure vLLM internals. Measured static-permit experiments showed that reducing gateway permits below 32 harms batching for the current workload.

The launcher mounts ${HF_CACHE_DIR:-$HOME/.cache/huggingface} into the container, refuses a floating latest image, and uses vLLM's generation defaults so a model repository cannot silently change benchmark sampling parameters.

After the server is healthy, run one declared load point with:

export MODEL=nvidia/Qwen3-8B-FP8
RUN_ID=fp8-low-rate REQUEST_RATE=0.5 NUM_PROMPTS=50 \
  ./scripts/run_vllm_baseline.sh

The benchmark uses fixed-length random-token requests, seeded open-loop arrivals, declared goodput SLOs, and writes its raw summary plus a Prometheus snapshot under results/raw/$RUN_ID.

After the low-load control passes, find the saturation knee with:

export MODEL=nvidia/Qwen3-8B-FP8
RATES="1.0 2.0 4.0 8.0" RUN_PREFIX=fp8-sweep \
  ./scripts/run_rate_sweep.sh

Every rate uses the same seed, prompt count, token lengths, and SLOs. The sweep prints a compact CSV from the saved vLLM summaries; raw per-request details remain available for paired analysis.

FCFS gateway

The M2 gateway implements a bounded, cancellation-safe FCFS permit queue in front of the pinned backend. It proxies streaming and non-streaming OpenAI-compatible completion endpoints, propagates request IDs, reports queue time, rejects a full admission queue with HTTP 429, and exports Prometheus metrics.

./scripts/build_gateway.sh
MAX_INFLIGHT=32 MAX_QUEUE_DEPTH=256 ./scripts/start_gateway.sh
curl -fsS http://127.0.0.1:8080/ready

Keep 32 permits for the current pinned workload; 16- and 24-permit experiments reduced throughput. The measured fixed admission setting is:

MAX_INFLIGHT=32 MAX_QUEUE_DEPTH=256 MAX_QUEUE_WAIT_SECONDS=0.75 \
  ./scripts/start_gateway.sh

Set MAX_QUEUE_WAIT_SECONDS to a positive value to reject requests that cannot enter the engine within a declared queue budget. The default is 0 (disabled), so admission behavior changes only when an experiment enables it explicitly.

Weighted-fair queueing and the tenant load client

QUEUE_POLICY=wfq orders the permit queue by start-time-fair virtual finish tags instead of arrival order; admission stays identical, so paired runs isolate queue ordering. Tenant weights and SLOs come from TENANT_POLICY_FILE; the calibrated prefill/decode cost model comes from SERVICE_COST_FILE.

QUEUE_POLICY=wfq \
TENANT_POLICY_FILE=config/tenants.example.toml \
SERVICE_COST_FILE=config/service-cost-fp8.json \
MAX_INFLIGHT=32 MAX_QUEUE_DEPTH=256 MAX_QUEUE_WAIT_SECONDS=0.75 \
  ./scripts/start_gateway.sh

The tenant-aware open-loop client sends exact token-id prompts, per-request tenant/token-count headers, and writes raw JSONL plus a per-tenant summary under results/raw/<run-id>/:

.venv/bin/python -m elasticserve.loadgen \
  --model nvidia/Qwen3-8B-FP8 --run-id my-run --seed 11 --rate 8 \
  --metrics-url http://127.0.0.1:8080/metrics

scripts/run_policy_pairs.sh replays the same seeds through FCFS and WFQ with alternating order and per-policy prompt salts; scripts/analyze_policy_pairs.py produces the paired tables, bootstrap intervals, and SLO-feasibility ceilings.

Repository map

elasticserve/
├── config/                 tenant-policy example
├── docs/                   architecture, revised plan, experiment protocol
├── scripts/                DGX baseline and environment capture
├── src/elasticserve/       policy, workload, metrics, simulator
└── tests/                  deterministic unit tests

Start with docs/project-plan.md, then execute the baseline gate in docs/experiment-protocol.md.