trtllm-serving

An end-to-end LLM serving stack for the NVIDIA DGX Spark (GB10): a FastAPI gateway in front of Triton Inference Server, running a compiled TensorRT-LLM engine.

The idea in three lines:

• Quantize and fuse offline, at build time. The engine is built once, so no request ever pays for compilation.
• Serve with Triton's in-flight batching and paged KV cache instead of one request at a time, so the GPU stays busy under load.
• Instrument everything with Prometheus and Grafana — latency percentiles, throughput, GPU utilization, and cost per token.

Every number in this README was measured on the box, and each one points at a committed artifact under results/. The headline, for Llama-3.1-8B-Instruct in NVFP4 on a single GB10:

• engine built offline in 68 s
• Triton ready 7 s after process start
• 1,485 output tokens/s at concurrency 64, p95 latency under 5 s
• $0.0315 per million output tokens at measured power draw

---

Architecture

flowchart LR
    LG["bench/loadgen.py<br/>closed-loop + Poisson open-loop"]

    subgraph HOST["DGX Spark (aarch64, GB10 sm_121, 128GB unified)"]
      subgraph COMPOSE["docker compose"]
        GW["FastAPI gateway :8080<br/>TTFT / TPOT / tokens / cost<br/>/metrics"]
        EXP["gpu_exporter :9835<br/>nvidia-smi + /proc/meminfo"]
        PROM["Prometheus :9090"]
        GRAF["Grafana :3000"]
      end
      subgraph TRITON["Triton Inference Server (own container, --gpus all)"]
        FE["OpenAI frontend :9000<br/>/v1/chat/completions + /metrics"]
        ENS["ensemble: preprocessing → tensorrt_llm → postprocessing"]
        BE["tensorrtllm C++ backend<br/>in-flight fused batching<br/>paged KV cache (32-token blocks)"]
      end
      ENGINE[("engines/llama31-8b-nvfp4-b16<br/>rank0.engine 5.8 GB<br/>built offline (68 s)")]
      GPU["GB10 GPU<br/>~231 GB/s measured"]
    end

    LG -->|"OpenAI SSE"| GW
    GW -->|"streaming passthrough"| FE
    FE --> ENS --> BE --> GPU
    BE -.->|"deserialize at startup (3.4 s)"| ENGINE
    GW -.->|"probe KV-cache metrics"| FE
    PROM -->|scrape| GW
    PROM -->|scrape| EXP
    PROM -->|scrape| FE
    GRAF --> PROM
    EXP -.->|"nvidia-smi, /proc/meminfo"| GPU

Loading

A few things worth knowing about the layout:

• Triton is not a compose service on purpose. It wants the whole GPU and gets restarted whenever I swap engines, and none of that should be coupled to restarting Prometheus.
• scripts/04_serve.sh runs Triton on the host network; the compose services reach it through host.docker.internal.
• The gateway is a measuring instrument, not a router. It streams bytes through untouched and records what a client actually experiences.

---

Build time, not request time

make build (scripts/03_build_engine.sh) produces the engine once, offline, inside the same tritonserver:25.10-trtllm-python-py3 image that serves it. That matters because engines are locked to the TensorRT-LLM version that built them — here 1.0.0 on TensorRT 10.11.

What the build does:

• Starts from NVIDIA's ModelOpt-quantized NVFP4 checkpoint (nvidia/Llama-3.1-8B-Instruct-FP4): E2M1 weights with a per-16-element FP8 scale, executed natively by Blackwell tensor cores.
• Fuses the graph into TensorRT plugins — fused attention with paged-context FMHA, fused GEMM + activation, fused norms — and picks kernels for sm_121 by timing them.
• Bakes in the runtime envelope: max_batch_size, max_num_tokens, max_seq_len.
• Serializes the result to engines/<name>/rank0.engine.

What that buys, measured:

• 68 s to build, 5.8 GB on disk.
• Triton startup is deserialization, not compilation: process start → all four models READY in 7 s, engine load 3.4 s (results/triton-coldstart.txt, from Triton's own log timestamps).
• The first request after startup costs 2.1 s against a 1.0 s steady state. That's tokenizer and CUDA-context warm-up in the Python pre/post-processing models — not kernel compilation.
• For contrast, the same checkpoint served through TensorRT-LLM's PyTorch backend (make serve-baseline, JIT autotuning at startup) takes 110 s to come up and pays a 1.8 s first-request JIT penalty (results/coldstart.txt).

The trade-off: batch size and sequence length are build-time constants. Changing max_batch_size means another 65 s build (MAX_BATCH_SIZE=64 make build) — which is exactly what the batch-size experiment below does.

Why NVFP4

Decode reads the full weight set once per token, and GB10 delivers ~231 GB/s measured. So the weight format sets the ceiling before any batching happens:

Format	Weight bytes	Single-stream decode ceiling
BF16	~16 GB	~14 tok/s
FP8	~8 GB	~29 tok/s
NVFP4	~4.5 GB	~51 tok/s

I use NVIDIA's published pre-quantized checkpoint rather than quantizing locally. Calibration quality dominates post-quantization accuracy, and their calibration sets are better than one assembled in an afternoon. scripts/02_quantize.sh is there for models with no published checkpoint.

---

In-flight batching and paged KV cache

Request-at-a-time serving leaves a decode-bound GPU idle between kernel launches — one 8B stream on this box is ~37 tok/s. Triton's tensorrtllm backend does two things about that:

• In-flight fused batching. Every iteration schedules whichever requests are ready, admitting new prompts into a running batch and retiring finished ones without draining.
• Paged KV cache. Each request's KV cache lives in 32-token blocks, so memory is allocated as sequences grow instead of reserved up front. With kv_cache_free_gpu_mem_fraction: 0.7 that is 59.3 GiB, 30,353 blocks, 971k tokens from a clean start, with block reuse across shared prefixes.

The model repository in triton/model_repository_engine/ sets batching_strategy: inflight_fused_batching, enable_kv_cache_reuse, enable_chunked_context, and decoupled streaming.

Here is what it buys on chat-short (128 in / 128 out, 100 requests per point, zero errors):

concurrency	batch-16 engine, tok/s	TTFT p50	batch-64 engine, tok/s	TTFT p50
1	37	40 ms	—	—
4	148	51 ms	—	—
8	280	52 ms	—	—
16	506	53 ms	490	167 ms
32	509	3.63 s	843	78 ms
64	—	—	1,485	151 ms

Reading the table:

• Concurrency 1 → 16 gives 13× the throughput at essentially the same TTFT. That's batching doing its job.
• At c=32 on the batch-16 engine, throughput is flat and TTFT jumps to 3.6 s. Every extra request is queue time, and Triton's scheduler metrics show it directly (nv_trt_llm_request_metrics, active vs scheduled).
• Rebuild for batch 64 and the same silicon serves 1,485 tok/s at c=64 — 40× a single stream — with p95 latency 4.6 s and TPOT still 35 ms.
• Peak KV-cache usage in these runs stayed under 2% of the pool. On this workload the batch cap binds long before memory does.

---

Instrumentation

Triton exports its own metrics, but the numbers a capacity study needs are what a client experiences. The gateway measures every request from the client's side of the wire:

Signal	How
TTFT	wall clock to the first SSE chunk carrying non-empty content; the role-only opening chunk is skipped
TPOT	(e2e − ttft) / (output_tokens − 1), one sample per request
Inter-token latency	one sample per chunk, so decode jitter is visible rather than averaged away
E2E latency	full request duration
Tokens in / out	Triton streams one token per chunk, so chunk counts are exact; stream_options.include_usage is used where an engine supports it
In-flight requests	observed concurrency — the right x-axis for a knee curve
Cost per token	amortized hardware price + measured electricity, attributed per request

Prometheus scrapes three sources every 5 s:

• The gateway — histograms with hand-picked LLM-latency buckets, so histogram_quantile returns a real p99 instead of interpolating inside a five-second bucket.
• The GPU exporter — utilization, SM clock, power, temperature, and the unified-memory picture from /proc/meminfo (because nvidia-smi's memory fields are [N/A] on Spark).
• Triton's /metrics — nv_inference_* core metrics plus the TRT-LLM backend's nv_trt_llm_kv_cache_block_metrics, nv_trt_llm_inflight_batcher_metrics, and nv_trt_llm_request_metrics.

Two details I care about:

• The gateway probes Triton's KV-cache block metrics and mirrors the used fraction as gateway_upstream_kv_cache_utilization. Availability is a separate gauge, so a missing metric never draws a flat zero line.
• The GPU exporter is its own process because nvidia-smi is a blocking subprocess call. Putting that in the gateway's event loop would inject stalls into the very TTFT measurements the gateway exists to take.

Grafana provisions a dashboard with latency percentiles, throughput, GPU telemetry, scheduler state, and $/1M tokens.

---

Measured results

Setup: DGX Spark (GB10, sm_121, 121 GiB unified LPDDR5X), driver 580.173.02, tritonserver:25.10-trtllm-python-py3 (TensorRT-LLM 1.0.0, TensorRT 10.11), nvidia/Llama-3.1-8B-Instruct-FP4. 2026-08-25 unless noted.

The knee, forward and reverse

Closed-loop, chat-short, batch-16 engine (results/20260825T123616Z-chat-short/, reverse order in results/20260825T125014Z-chat-short/):

c	tok/s (fwd)	tok/s (rev)	TTFT p50	TPOT p50	e2e p95	$/1M out
1	37.4	38.1	40 ms	26.7 ms	3.48 s	1.25
2	75.3	75.2	52 ms	26.4 ms	3.41 s	0.62
4	148.3	147.9	51 ms	26.8 ms	3.47 s	0.32
8	280.2	279.8	52 ms	27.3 ms	3.54 s	0.17
16	505.5	506.3	53 ms	28.3 ms	3.67 s	0.092
32	509.2	510.8	3.63 s	28.2 ms	7.30 s	0.092

• The reverse sweep reproduces every point to within 2%, so the knee is the batch cap, not thermal drift.
• Against the JIT PyTorch path on the same checkpoint, the compiled engine is faster single-stream (37 vs 27–38 tok/s) and at first token (53 vs 76 ms at c=16), and ~9% behind on saturated batch-16 throughput (506 vs 554 tok/s). The batch-64 engine more than recovers that (1,485 vs 1,395 tok/s).
• Power under load: 49 W p90 from the GPU exporter, versus 62 W on the PyTorch path. Cost figures use it with $3,999 amortized over three years and $0.18/kWh.

Open loop past the knee

Poisson arrivals at fractions of the knee rate (λ* = 3.95 req/s), 120 s windows, latency counted from scheduled arrival. Generator dispatch delay p99 stayed ≤ 2.4 ms in every run (results/*open-loop/):

offered	achieved	e2e p50	e2e p99	observed concurrency
0.5×	1.79/s	3.61 s	3.70 s	6.5
0.9× (seed 1234)	3.40/s	3.74 s	4.86 s	13.1
0.9× (seed 777)	3.51/s	3.73 s	5.44 s	13.8
1.0×	3.72/s	3.79 s	5.82 s	15.2
1.2×	4.15/s	7.28 s	15.2 s	34.7

• Below the knee, latency is just service time, and Little's Law holds exactly: at 0.5×, L = 1.79 × 3.61 = 6.45, observed 6.45.
• At 1.2× the queue grows without bound. Achieved throughput stalls below the offered rate and 35 requests sit in flight against 16 slots — and Triton's own request_type="active" gauge peaked at 36 over the same window.
• Closed-loop c=16 reported e2e p95 of 3.67 s. Open-loop at the same offered rate reports p99 of 5.8 s. Closed-loop answers "how fast can 16 patient users go?"; open-loop answers what the 99th-percentile user sees when arrivals don't wait for departures — which is the number capacity planning actually needs.

Workload sensitivity

All four workloads knee at c=16 — the batch cap binds regardless — but throughput at the knee tells the prefill story (results/20260825*-{chat-long,summarize,rag}/, zero errors throughout):

workload	in / out tokens	tok/s at knee	TTFT p50 at knee	e2e p95	$/1M out¹
chat-short	128 / 128	506	53 ms	3.67 s	0.092
chat-long	256 / 1024	468	69 ms	31.8 s	0.100
summarize	4096 / 256	316	94 ms	12.4 s	0.148
rag	6144 / 128	228	430 ms	9.17 s	0.205

¹ Measured 49 W GPU power, $3,999 over 3 years, $0.18/kWh.

• Prefill is where the compiled 1.0.0 engine gives ground to the newer PyTorch path: rag first-token at c=16 is 430 ms here versus 166 ms on TensorRT-LLM 1.3's kernels, and rag throughput is 228 versus 278 tok/s.
• Decode-dominated workloads land within 6–10% of the PyTorch path.
• Even so, the engine chews through ~11,000 prompt tokens/s on rag at the knee, alongside its 228 output tokens/s.

Chunked context (CHUNKED_CONTEXT=false on make serve for the A/B, results/*rag-nochunk/): with max_num_tokens=8192, a 6,144-token prompt fits in one iteration, so chunking only changes how prefill interleaves with running decodes. On rag at c=16 it costs TTFT — 430 ms chunked versus 267 ms unchunked — at the same 228–232 tok/s and an identical 64 ms decode TPOT. The prompt gets split into pieces scheduled across iterations, and the running decodes gain nothing from it. Chunked context is the right default when prompts exceed the per-iteration token budget; when they don't, it's latency for nothing.

Accuracy through the serving path

GSM8K, full 1,319-question test set, evaluated end-to-end through the Triton endpoint (lm-eval-harness gsm8k_cot_llama, chat template, 8-shot multiturn, greedy; results/gsm8k-triton.txt):

• 70.8% strict-match ± 1.3% (71.7% flexible) through the compiled engine.
• 78.6% for the same NVFP4 checkpoint through the TensorRT-LLM 1.3 PyTorch backend, scored with the identical harness configuration.
• 84.5% is Meta's published number for BF16.

The 8-point gap between the two serving paths is the compiled 1.0.0 engine's kernels — the NVFP4 GEMM and attention implementations TensorRT 10.11 selects for sm_121 — not the quantized weights. That's the real price of the compiled-engine path on this hardware today, alongside its faster startup and lower power.

Getting a true number meant stepping around three measurement traps, each of which first produced a dramatic fake "quantization loss":

• Raw-completions evaluation without the chat template scored 26%.
• A 256-token generation budget truncated chain-of-thought answers before the final number.
• The default extraction regex grabbed the first operand of "The answer is 9 * $2 = $18" instead of 18.

If a quantized endpoint scores catastrophically, suspect the harness before the weights.

The page cache eats the KV cache

This is the most Spark-specific behaviour in the repo. GPU and CPU share one 128 GB pool, and cudaMemGetInfo cannot see that clean page-cache pages are reclaimable. Three starts of the same engine and configuration (results/triton-pagecache.txt):

page cache at start	paged KV cache allocated
3.4 GiB (after drop_caches)	59.3 GiB / 971,296 tokens
~20 GiB (after the morning's builds)	40.1 GiB / 657,120 tokens
71.8 GiB (after a large download)	15.4 GiB / 252,864 tokens

No error, no warning — up to 3.8× less KV capacity, which on long-context workloads shows up as a lower concurrency ceiling that looks like a scheduler problem. The PyTorch path shows the same effect (32.0 GB vs 63.9 GB). So:

sync && echo 3 | sudo tee /proc/sys/vm/drop_caches   # before make build or make serve

make preflight warns when the page cache exceeds 8 GiB; make preflight-fix drops it.

---

Running it

On a DGX Spark

git clone <this repo> && cd trtllm-serving

make preflight          # arm64? GPU runtime (nvidia runtime or CDI)? page cache? disk? ports?
make preflight-fix      # ...and drop the page cache (needs sudo)
make pull               # tritonserver:25.10-trtllm-python-py3, verified arm64 with trtllm-build
make build              # compile engines/llama31-8b-nvfp4-b16 (68 s)
make serve              # Triton + engine on :9000 (blocks)

In a second terminal:

make up                 # gateway + gpu-exporter + prometheus + grafana
make warmup             # absorb the first-request warm-up before measuring
make bench              # concurrency sweep 1..32 -> results/*.jsonl + charts
make analyze            # percentile tables + knee curve + cost per 1M tokens

Grafana provisions its dashboard at http://localhost:3000 (admin/admin).

Knobs and alternatives:

• MAX_BATCH_SIZE, MAX_NUM_TOKENS, MAX_SEQ_LEN on make build — they are baked into the engine.
• ENGINE=<name> and CHUNKED_CONTEXT=false on make serve.
• make serve-baseline — TensorRT-LLM's PyTorch backend via trtllm-serve, JIT at startup.
• make serve-llmapi — Triton with the PyTorch backend through the llmapi model.
• ENGINE_URL=... make up points the gateway at whichever engine is running.

Without a GPU

pip install -r requirements-dev.txt
make test               # 66 tests, no GPU, no network, no container

make mock               # fake engine on :9000 with configurable TTFT/ITL
make dev-gateway        # gateway on :8080 pointed at the mock
python bench/loadgen.py --base-url http://127.0.0.1:8080 --mode closed \
    --concurrency 4 --num-requests 20 --no-force-output-length

bench/mock_server.py sleeps for its simulated prefill after emitting the role-only SSE chunk — the same sequence a real engine produces — and the test suite asserts the gateway's TTFT clock doesn't start on that first, contentless chunk.

---

Spark operational notes

• Memory queries return [N/A]. No discrete VRAM; the honest picture is /proc/meminfo (MemAvailable), which the exporter publishes as dgx_unified_memory_*_bytes.
• Drop the page cache before building or serving. See above — it silently shrinks the paged KV cache.
• The TensorRT builder needs the pool too. make build refuses to run while a serving container holds the GPU.
• arm64 image discipline. docker pull will happily fetch an x86_64 manifest; scripts/01_pull_container.sh verifies .Architecture == arm64 and that trtllm-build and the tensorrtllm backend are present.
• Version lock. The compiled-engine path needs a Triton image whose TensorRT-LLM still ships the TensorRT backend (≤ 1.1; 25.10 bundles 1.0.0). Later images (TensorRT-LLM 1.2+) build no engines.
• GPU-in-Docker works via CDI on current toolkits — no registered nvidia runtime required; preflight accepts either.
• No clock locking. SM clock floats 208–2,548 MHz with load. Warm up first and compare like against like.

Repository map

config.yaml            single source of truth: model, engine URL, cost model, loadgen defaults
docker-compose.yml     gateway + gpu-exporter + prometheus + grafana (NOT Triton)
Dockerfile             small pure-Python image for the gateway and exporter
ruff.toml              lint config (vendored triton/ excluded)

scripts/
  00_preflight.sh      arm64, driver, GPU runtime/CDI, page-cache pressure, disk, ports
  01_pull_container.sh pull the Triton + TensorRT-LLM image, verify arm64 and the engine toolchain
  02_quantize.sh       NVFP4 PTQ via Model Optimizer (only for models with no published checkpoint)
  03_build_engine.sh   offline engine build (quantized checkpoint in, fused engine out)
  build_engine.py      the build itself, run inside the Triton image
  04_serve.sh          Triton + compiled engine + OpenAI frontend
  05_warmup.sh         absorb the first-request warm-up before measuring
  06_run_bench.sh      concurrency sweep + environment capture + analysis
  08_serve_triton_llmapi.sh  comparison: Triton with the TRT-LLM PyTorch backend (llmapi model)
  09_serve_trtllm_baseline.sh  baseline: trtllm-serve, PyTorch backend, JIT at startup

triton/
  model_repository_engine/   inflight_batcher_llm ensemble (preprocessing, tensorrt_llm, postprocessing)
  model_repository_llmapi/   llmapi model for the PyTorch-backend comparison
  frontend/triton.py         vendored OpenAI-frontend request builder (+ min_tokens, token counts)

gateway/
  main.py              FastAPI app: streaming passthrough, /health, /ready, /metrics, KV probe
  metrics.py           histogram definitions with LLM-scaled bucket edges
  cost.py              cost model (unit tested against hand-computable numbers)
  upstream.py          httpx streaming client + SSE parsing + TTFT logic
  config.py            pydantic settings: YAML file, overridden by env

exporter/gpu_exporter.py   nvidia-smi + /proc/meminfo Prometheus exporter

bench/
  loadgen.py           closed-loop and open-loop (Poisson) async load generator
  analyze.py           JSONL -> percentile tables, knee curve, cost charts
  mock_server.py       fake engine for GPU-free development
  workloads.yaml       chat-short, chat-long, summarize, rag

monitoring/            prometheus.yml + Grafana provisioning + dashboard
engines/               compiled engines (gitignored; rebuild with make build)
results/               committed benchmark runs, metrics dumps, and environment captures
tests/                 66 pytest tests, all GPU-free

Cost model caveat

Cost numbers are only as honest as config.yaml:

• set cost.hardware_price_usd to what you paid,
• set cost.duty_cycle to the fraction of the box's life it will actually serve traffic,
• pass measured wattage to bench/analyze.py --power-watts (the exporter gives it to you) rather than trusting the default.