Skip to content

README.md

Latest commit

 

History

History
247 lines (178 loc) · 15.2 KB

README.md

File metadata and controls

247 lines (178 loc) · 15.2 KB
Raw

Benchmarking Report

A benchmarking report is a standard data format describing the cluster configuration, workload, and results of a benchmark run. The report acts as a common API for different benchmarking experiments. Each supported harness in llm-d-benchmark creates a benchmark report upon completion of a run, in addition to saving results in its native format.

There are three versions of the benchmark report, 0.1, 0.2, and 0.2.1. All reports are generated by the llm-d-benchmark harness pod, but new applications consuming benchmark data should use version 0.2.1 reports. Version 0.2.1 is an additive superset of 0.2: every valid 0.2 report is also a valid 0.2.1 report.

v0.2 Format Description

A benchmark report describes the inference service configuration, workload, and performance results. Individual traces from single inference executions are not captured, but time-series or statistical results may be captured.

See br_v0_2_example.yaml for a dummy example report (values may be non-sensical). A JSON Schema for the benchmark report version 0.2 is at br_v0_2_json_schema.json.

The top-level fields for the benchmark report are version, run, scenario, and results.

version Field

The version field (0.2) indicates the schema revision for the benchmark report. Different versions may add or drop certain details, so there is no guarantee of "lossless" conversion between benchmark report versions.

run Field

The run field contains details about the benchmark run. Only the unique ID run.uid is required, with other fields including start/end times and duration, user name, and additional IDs for the experiment (to group benchmark reports), cluster, and pod.

scenario Field

scenario describes the inference stack configuration (under scenario.stack) and workload details (under scenario.load). These are the inputs to a benchmark experiment, and ideally will be sufficiently populated to ensure the benchmark is reproducible and leaves no question as to what exactly is being measured.

scenario.stack consists of a list of components such as inference engine instances (vLLM, SGLang...), routers, and any other part of the system used to support the inference endpoint. No specific component is required, so this may be customized to the needs of a particular experiment.

Each component has a metadata, standardized, and native field. The native subsection can store any arguments, environment variables, and configuration details in the native format of a particular component. The standardized subsection stores details in a single standardized schema for that component, regardless of the specific vendor or version used. For example, an inference engine component could be vLLM, SGLang, etc. but they will always certain configuration options in common albeit defined in different ways. The standardized subsection for an inference engine contains common details like LLM model name, accelerator hardware model and count, and parallelisms used.

The workload details under scenario.load follow the same methodology as components, with metadata, standardized, and native subsections.

results Field

The results field contains outputs from a benchmarking experiment. This may include any one of request-level performance metrics, session-level performance metrics, observability data, stack component health, or profiling.

results.request_performance

Aggregate statistics for individual inference requests: latencies (TTFT, TPOT, ITL, E2E), throughput (input/output/total token rates, request rate), and request counts.

results.session_performance

Session-level statistics for multi-turn workloads. Populated from *_session_lifecycle_metrics.json files produced by inference-perf when multi-turn sessions are enabled. The session_performance.sessions object contains:

Field Description
total Total number of sessions run
succeeded / failed Session outcome counts
total_events / total_events_completed / total_events_cancelled Event (per-turn request) counts
session_rate Rate of session completions per second (Statistics)
session_duration Distribution of session wall-clock durations in seconds (Statistics)
events_per_session Distribution of event counts per session (Statistics)
events_cancelled_per_session Distribution of cancelled event counts per session (Statistics)
input_tokens_per_session Distribution of total input tokens per session (Statistics)
output_tokens_per_session Distribution of total output tokens per session (Statistics)

A session_performance report is generated alongside the standard request_performance report for each stage that has a corresponding session lifecycle file. The scenario.load.standardized.stage field identifies which stage the report covers, and scenario.load.standardized.multi_turn.enabled is set to true.

v0.2.1 Format Description

Version 0.2.1 is an additive minor revision of 0.2 that adds optional multi-modal payload statistics for image, video, and audio workloads. Every field introduced is optional, so any 0.2 report validates unchanged under 0.2.1 (enforced by tests/test_benchmark_report_v0_2_1_compat.py).

See br_v0_2_1_example.yaml for a dummy example report, and br_v0_2_1_json_schema.json for its JSON Schema. All other fields and sections are identical to 0.2.

The additions, all derived from what the client can determine from the payloads it sent, are:

  • results.request_performance.aggregate.requests.request_size (Statistics): total encoded request size in bytes, capturing the large payloads typical of multi-modal requests.
  • results.request_performance.aggregate.requests.multimodal: a per-modality block (image, video, audio), each a distribution set over the media instances sent. image/video carry count, filesize, pixels, and aspect_ratio; video adds frames; audio carries count, filesize, and duration.
  • results.request_performance.aggregate.throughput.{image,video,audio}_rate (Statistics): per-modality delivery rates (images/s, videos/s, audios/s).

New unit categories back these fields: pixels (quantity), ratio (for aspect ratio, distinct from a 0..1 portion), bytes (memory), and a media-throughput category (images/s, videos/s, audios/s) kept separate from the request-rate category so the existing request_rate guardrail is unaffected.

Server-side multi-modal metrics (vision token counts, encoding time, multimodal cache hit rates) are out of scope for this revision.

v0.1 Format Description

A benchmark report describes the inference service configuration, workload, and aggregate results. Individual traces from single inference executions are not captured, rather statistics from multiple traces of identical scenarios are combined to create a report.

See br_v0_1_example.yaml for a dummy example report (values may be non-sensical). The JSON Schema for the benchmark report version 0.1 is at br_v0_1_json_schema.json. The report has three top-level fields, version, scenario, and metrics.

While each of these fields is required, some subfields may be optional or not apply to the specific benchmark being performed. For example, some metrics may not be captured or supported by a certain benchmarking toolset. In cases where one desires to capture information that is not part of the standard benchmark report schema, a metadata field may be placed almost anywhere under scenario or metrics to add arbitrary data.

version Field

The version field (0.1) indicates the schema revision for the benchmark report. Different versions may add or drop certain details, so there is no guarantee of "lossless" conversion between benchmark report versions.

scenario Field

The scenario describes precisely what was measured. This includes details about the inference platform (full stack, including versions and the important runtime arguments), cluster configuration (like GPUs and parallelism utilized), and workload. The content in this field should be detailed enough that it is sufficient to launch a repeat benchmarking experiment that will yield similar results (within some reasonable bound of variability).

Note

With future revisions the scenario field could be used as an input format for executing benchmark runs. For this to be practical we would first need to standardize the definition of a workload, specifically in scenario.load.args which is currently the exact arguments (non-standard) sent to a particular harness. In addition, the input format would need a way to describe swept parameters. A benchmark report currently describes a single point in any sweep.

metrics Field

The metrics field contains all of the results for the report. This does not include individual trace details, rather statistics for all runs that were captured in benchmarking for a particular scenario. This includes request-level performance metrics (like latencies and throughput), details about the inference service (like request queue lengths and KV cache size), and hardware metrics (such as GPU compute and memory utilization). Some of the underlying fields, such as the hardware metrics, more traditionally fall in the category of "observability" rather than "benchmarking".

Implementation and Usage

The schema for a benchmarking report is defined through Python classes using Pydantic in schema_{version}.py. Instantiating an instance of a benchmark report class includes various checks, such as ensuring compliance with the schema, proper use of units, and defining all required entities.

Requirements

numpy>=2.3.1
pydantic>=2.11.7
PyYAML>=6.0.2

Creating a BenchmarkReport

A BenchmarkReport class may be instantiated from a JSON/YAML file with import_benchmark_report(), or from a JSON/YAML string with yaml_str_to_benchmark_report(). Both of these functions will detect the appropriate version of the benchmark report.

To generate a JSON or YAML string from a BenchmarkReport instance, use the get_json_str() and get_yaml_str() methods. To save as a JSON/YAML file, use the export_json() or export_yaml() methods.

import benchmark_report

# Instantiate a BenchmarkReport instance from a YAML file
br=benchmark_report.import_benchmark_report("benchmark_report/br_v0_2_example.yaml")

# Create a YAML string from the BenchmarkReport
br_yaml_str = br.get_yaml_str()

# Print the string
print(br_yaml_str)

# Save the BenchmarkReport as a YAML file
br.export_yaml("br_test.yaml")

An instance of BenchmarkReport may be created directly, for example:

import benchmark_report

br = benchmark_report.BenchmarkReportV02(**{
    "run": {
        "uid": "38b1f169ca178b756f7483523b17de61",
    },
    "scenario": {
        "stack": [
            {
                "metadata": {
                    "schema_version": "0.0.1",
                    "label": "vllm-svc-0",
                    "cfg_id": "cc73fc6b51a1d3b8128f312d70476d7c",
                },
                "standardized": {
                    "kind": "inference_engine",
                    "tool": "llm-d",
                    "tool_version": "ghcr.io/llm-d/llm-d-cuda:0.3.1",
                    "role": "decode",
                    "replicas": 2,
                    "model": {
                        "name": "Qwen/Qwen3-0.6B",
                    },
                    "accelerator": {
                        "model": "NVIDIA-H100-80GB-HBM3",
                        "count": 8,
                        "parallelism": {
                            "tp": 8,
                        },
                    },
                },
                "native": {},
            },
        ],
        "load": {
            "metadata": {
                "schema_version": "0.0.1",
                "cfg_id": "a4e18f265cc33786a42b8a3f7ac2edcb",
            },
            "standardized": {
                "tool": "inference-perf",
                "tool_version": "0.3.0",
                "input_seq_len": {
                    "distribution": "fixed",
                    "value": 2148,
                },
                "source": "sampled",
            },
            "native": {},
        },
    },
    "results": {
        "request_performance": {
            "aggregate": {
                "latency": {
                    "request_latency": {
                        "units": "s",
                        "mean": 0.0368578234128654,
                    },
                },
            },
        },
    },
})

print(br.get_yaml_str())

The native formats returned by different harnesses may be converted to a benchmark report using functions in native_to_br0_2.py, or using the CLI defined in cli.py.

To run the converter CLI, execute the module using python -m from the root of this repository:

python3 -m llmdbenchmark.analysis.benchmark_report.cli \
  --workload-generator <generator_name> \
  --br-version <version> \
  <path_to_raw_results_file> \
  [path_to_output_report_yaml]

Parameters Reference

  • -w, --workload-generator: Specifies the harness generator. Must be one of: 'guidellm', 'inferencemax', 'inference-perf', 'vllm-benchmark', 'nop'.
  • -b, --br-version: Target benchmark report version (defaults to 0.1; use 0.2 for the standard version, or 0.2.1 to additionally capture the multimodal payload statistics that inference-perf emits).
  • -f, --force: Overwrites the output file if it already exists.
  • results_file (Positional): Path to the raw native results file to convert. (e.g. For inference-perf, this must contain "stage_" in its filename, e.g., stage_0_lifecycle_metrics.json).
  • output_file (Positional, Optional): Destination for the converted report. If omitted, the YAML output is printed directly to stdout.

Example

To convert a raw stage_0_lifecycle_metrics.json results file from an inference-perf run and save the v0.2 report to benchmark_report_v02.yaml:

python3 -m llmdbenchmark.analysis.benchmark_report.cli \
  -w inference-perf \
  -b 0.2 \
  ./stage_0_lifecycle_metrics.json \
  ./benchmark_report_v02.yaml

Session Lifecycle Files (Inference-Perf)

For inference-perf session lifecycle files (*_session_lifecycle_metrics.json), pass the -s / --session flag:

# Print to stdout
python3 -m llmdbenchmark.analysis.benchmark_report.cli -w inference-perf -b 0.2 -s stage_0_session_lifecycle_metrics.json

# Save to file
python3 -m llmdbenchmark.analysis.benchmark_report.cli -w inference-perf -b 0.2 -s stage_0_session_lifecycle_metrics.json benchmark_report_session.yaml

The -s flag selects the import_inference_perf_session converter, which populates results.session_performance instead of results.request_performance. The inference-perf-analyze_results.sh script detects *_session_lifecycle_metrics.json files automatically and passes -s without manual intervention.