LLM Batching Energy

This project studies the tradeoff between GPU efficiency and CPU scheduling overhead in LLM inference systems.

We compare static batching and continuous batching under different request workloads.

The current stage of the project is local scaffolding. The goal right now is to make the repository coherent and runnable locally with mock execution before adapting it to a Linux GPU VM and real vLLM serving.

Setup

Create a virtual environment and install the local dependencies:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Notes:

• psutil improves local CPU and memory sampling, but the runner has a fallback if it is missing.
• matplotlib is required for plot generation. Without it, analysis still writes results/summary.csv and skips plots.
• GPU metrics are optional and appear only when nvidia-smi is available.

Hypothesis

Continuous batching improves GPU utilization, but under high request variability, CPU scheduling overhead reduces overall system efficiency.

Experiment Design

We evaluate four conditions:

1. Static batching + low variance requests
2. Static batching + high variance requests
3. Continuous batching + low variance requests
4. Continuous batching + high variance requests

Metrics

We measure:

• Throughput (tokens/sec)
• Latency
• GPU utilization, GPU memory, and GPU power (nvidia-smi), sampled on a background interval during real runs
• Host-wide CPU and memory (psutil), plus RSS and summed CPU% for processes whose command line matches patterns (default: vllm) to approximate the serving engine cost
• Runner (experiment client) CPU and RSS
• Energy:
  • GPU: trapezoid integration of reported GPU power over the sampling interval (summary.energy.gpu_energy_joules_integrated)
  • CPU / memory subsystem (Linux): Intel/AMD RAPL energy from /sys/class/powercap when readable (summary.energy.rapl_*)
  • Total (when both sides are available): summary.energy.total_measured_energy_joules ≈ RAPL (package + DRAM + other classified zones) + integrated GPU energy. summary.estimated_energy_joules mirrors this when measured; otherwise it falls back to average GPU power × makespan, or (mock only) the toy model.

Install psutil (pip install -r requirements.txt) so host and server-process metrics are populated. On Linux, RAPL files may require appropriate permissions (sometimes root or CAP_SYS_RAWIO); if RAPL is unreadable, CPU-side Joules stay null while GPU metrics still work.

GPU-specific metrics are unavailable without NVIDIA drivers. Fields remain optional on macOS or CPU-only hosts.

Current Scope

This repository is intended to become an experiment pipeline that can:

• load experiment configs from experiments/*.json
• generate workloads reproducibly
• run local mock experiments first
• log runtime and system metrics
• save structured results in results/
• aggregate repeated runs
• generate plots in plots/

It is not yet intended to optimize for production deployment, distributed serving, or cloud-specific infrastructure.

Experiment Config Schema

Each experiment file in experiments/ defines one condition. The current schema is:

• name: short stable identifier for the condition
• batching: static or continuous
• variance: low or high
• num_requests: number of requests to generate
• seed: random seed for reproducible workload generation
• repeats: number of times the condition should be run
• prompt_length: prompt length distribution spec
• output_length: output length distribution spec

Length distributions use an explicit object form:

• fixed length:

{"type": "fixed", "value": 50}

• uniform range:

{"type": "uniform", "min": 10, "max": 200}

Run outputs now use a stable research-friendly layout:

• results/<condition>/<timestamp>/config.json
• results/<condition>/<timestamp>/summary.json
• results/<condition>/<timestamp>/run_001.json (one file per repeat)
• optional workload snapshots:
  • shared mode: results/<condition>/<timestamp>/workload.json
  • per-repeat mode: results/<condition>/<timestamp>/workload_r001.json

Aggregated outputs:

• results/summary.csv
• plots/*.png

Local Workflow

Phase 3 adds a local mock execution path on top of deterministic workload generation.

Run one config with its configured repeats:

python src/run_experiment.py --config experiments/static_low.json --save-workload

Run all conditions in one command:

python src/run_experiment.py --config-dir experiments --save-workload

Install GPU static-batching deps when needed:

pip install -r requirements-hf-static.txt

Static vs continuous batching (real runs)
vLLM does not turn off continuous batching. This repo matches the usual lab setup:

• --backend hf_static — true static batching: Hugging Face generate() once per batch, left-padded prompts, shared max_new_tokens (longest output in the batch). Use with configs where batching is static only.
• --backend vllm — continuous batching: HTTP to a running vLLM server. Use with configs where batching is continuous only.
• --backend hybrid — runs all static_*.json and continuous_*.json in one experiments/ pass: static conditions use HF, continuous use vLLM. Requires both --hf-model and --vllm-model. Static conditions are run first; the HF weights are dropped from GPU memory before any vLLM condition so the vLLM process can own the card.

Continuous-only (vLLM server must be up):

python src/run_experiment.py \
  --config-dir experiments \
  --backend vllm \
  --vllm-endpoint http://127.0.0.1:8000 \
  --vllm-model <same-id-as-server> \
  --vllm-max-concurrency 8 \
  --replay-time-scale 0.0

Full factorial static + continuous (HF for static, vLLM for continuous):

python src/run_experiment.py \
  --config-dir experiments \
  --backend hybrid \
  --hf-model <hf-model-id> \
  --hf-static-batch-size 8 \
  --vllm-endpoint http://127.0.0.1:8000 \
  --vllm-model <vllm-model-name> \
  --vllm-max-concurrency 8 \
  --replay-time-scale 0.0

Static-only (no vLLM process):

python src/run_experiment.py \
  --config-dir experiments \
  --backend hf_static \
  --hf-model <hf-model-id> \
  --hf-static-batch-size 8

Notes for minimal real-backend runs:

• --backend mock preserves local simulation behavior.
• --monitor-interval-s (default 0.25): background sampling for host/GPU/RAPL. Must be > 0 for vllm, hf_static, and hybrid. Use --monitor-interval-s 0 with --backend mock only.
• --server-cmdline-pattern applies to vLLM monitoring; HF static runs inference in-process (patterns cleared for that path).
• Use the same base model for --hf-model and the vLLM server when you want a fair comparison.
• estimated_energy_joules prefers RAPL + integrated GPU when available; otherwise GPU average power × makespan.

Use dry-run validation without writing files:

python src/run_experiment.py --config-dir experiments --dry-run

Run a subset of repeats (1-based, inclusive):

python src/run_experiment.py --config experiments/static_low.json --repeat-start 1 --repeat-end 2

Set a custom output root and run label:

python src/run_experiment.py --config-dir experiments --results-dir results --run-label pilot

To generate only the workload snapshot without running the mock scheduler, use:

python src/run_experiment.py --config experiments/static_low.json --generate-only

To inspect the generated JSON payload directly, use:

python src/run_experiment.py --config experiments/static_low.json --stdout

For local scaffolding, generated requests arrive at 1-second intervals in generation order.

Workload strategy for repeats:

• default behavior is per_repeat: each repeat uses a derived seed (seed + repeat_index - 1) so repeats are deterministic but not identical.
• optional --reuse-workload: generate one workload and reuse it across repeats for direct scheduler-only comparison.

The mock execution mode is intentionally simple:

• static groups requests into fixed-size batches and uses the longest request in each batch to determine batch duration
• continuous uses a small slot-based scheduler that starts requests as capacity opens

These mock semantics are placeholders for local development. They are useful for testing the pipeline and output structure, not for drawing real system conclusions.

Explicit Mock Limits

• static is a local approximation using fixed-size grouping and longest-request padding.
• continuous is a local slot-based approximation and not a faithful serving runtime.
• GPU metrics are optional and populated only when nvidia-smi is available.
• Energy is currently an estimate derived from timing and CPU/memory proxies.

Treat local mock outputs as pipeline-validation artifacts, not final evidence.

Batching Semantics

At this stage:

• continuous means the condition is intended to represent dynamic batching behavior and should later map to vLLM-style serving
• static means the condition is intended to represent a simple, explicit, reproducible approximation of fixed batching

Those execution details are implemented in the local mock runner first, then adapted to real serving later.

Analysis Workflow

After generating runs, aggregate metrics and build plots:

python src/analyze_results.py --results-dir results --summary-csv results/summary.csv --plots-dir plots

Analyze only the latest invocation directory per condition:

python src/analyze_results.py --results-dir results --latest-only

Include older flat root-level artifacts if you want backward-compatible aggregation:

python src/analyze_results.py --results-dir results --include-legacy-flat

The analyzer reports condition-level averages across repeats for:

• throughput
• average latency and p95 latency
• average queue time
• CPU and memory with separate measured, estimated, and effective columns
• estimated energy
• GPU utilization/memory/power (nullable until GPU-capable environment)

Validation Workflow

Run a clean local validation pass across all experiment configs:

python src/validate_local.py --clean --save-workload

This command:

• runs the mock experiment pipeline for all configs
• writes validation artifacts under results/local_validation/
• writes a validation summary CSV
• writes plots under plots/local_validation/ when matplotlib is installed

Structure

• src/ – scripts for generating workloads, running experiments, monitoring, and analysis
• experiments/ – config files for each experiment condition
• results/ – structured run artifacts plus summary.csv
• plots/ – graphs generated from results
• notes/ – planning and research notes
• draft/ – paper writing

Goal

Understand when continuous batching is truly more efficient, and when its overhead outweighs its benefits.