Throughput statistics for min/max/stdev/percentiles do not give a good representation of the benchmark

[DenDiv]

Bug Description

Description

I am benchmarking Llama 3.1 8B on an NVIDIA H100 GPU. When running in concurrent mode, I observed highly inconsistent metrics:

• A significant discrepancy between Mean and Median RPS and TPUT metrics.
• The Standard Deviation (std) for the RPS metric reached ~900, even though the actual maximum RPS was around 50.

Observations

• Console output:

• Data file: concurrenct50.json

Debug

I suppose the problem is caused by a very low threshold for merging. In concurrent mode, many requests can finish almost simultaneously, which leads to durations being as low as 1e-4 in this line, resulting in an inflated rate. I set the threshold to 1.0 and received more stable results.

Expected Behavior

Stable quantile metrics and low variance

Steps to Reproduce

Below code for reproducing problem: start model in sglang and benchmark scenario

model start:

python3 -m sglang.launch_server \
  --model-path /models/Llama/Llama3.1-8B-Instruct/ \
  --tp=1 \
  --dp=1 \
  --enable-metrics \
  --disable-radix-cache \

scenario.json:

{
  "profile": "concurrent",
  "rate": 50,
  "max_seconds": 10,
  "target": "http://localhost:30000",
  "data": "prompt_tokens=128,output_tokens=128",
  "processor": "/models/Llama/Llama3.1-8B-Instruct/",
}

Operating System

Ubunty 22.04

Python Version

Python 3.12.12

GuideLLM Version

guidellm version: 0.6.0.dev75

Installation Method

pip install guidellm

Installation Details

No response

Error Messages or Stack Traces

Additional Context

No response

[sjmonson]

Sorry for the delay, this has been a bit of a rabbit-hole to answer correctly. The problem with what you did is by increasing the threshold you are really just smoothing the data. A way to interpret threshold=1 is that you are sampling tokens over 1 second and treating it as if they have all arrived at the end of that second. If there is any instability within that second you won't see it in the data.

Compare fixed-1s (threshold=1) to main (threshold=1e-4). Main shows that there is a skew away from the mean into lower throughput. This is likely correlated to first token processing periods, since concurrency launches all initial requests at the same time there is an expected pattern of low output throughput at the launch of requests. In comparison the fixed-1s distribution has completely lost its shape, all of the data has been condensed.

[DenDiv]

@sjmonson Thanks for the clarification. I’ve investigated further and agree that threshold = 1.0 simply smoothes the data rather than being a bug in guidellm.

However, I'm not sure if the approach used in the rate_distribution_from_timings function is appropriate for calculating statistics when using a non-constant profile. Please see the rates values for the RPS metric below when comparing constant and poisson profiles.

scenario:

{
  "profile": "constant", // or poisson 
  "rate": 30,
  "max_seconds": 60,
  "target": "http://localhost:30000",
    "data": {
      "prompt_tokens": 128,
      "output_tokens": 128
  },
  "sample_requests": 20,
  "output_dir": "/qwarium/home/d.divitskiy/const_vs_conc",
  "processor": "/models/Llama/Llama3.1-8B-Instruct/"
}

When using a constant profile, everything works as expected. rate_distribution_from_timings will correctly highlight any rate instability within a second:

When using poisson profile, the rate values can fluctuate wildly — becoming very high (when requests arrive at the minimum interval equal to the 1e-4 threshold) or very low:

While _rate_distribution_from_timings works well for the constant scenario, a different approach might be better for other profile types where variability is expected by design.

[sjmonson]

Yeah the current strategy is definitely most ideal for constant rps since its a very homogeneous workload. Poisson and concurrent are more wave-like patterns which is hard to represent with min/max/stdev/percentiles. We have debated in the past evenly spacing concurrent requests during startup which would alleviate some of this issue. However we ultimately decided against it because most other LLM load testing tools implement concurrent in a similar manner and the bursty-ness is useful for exposing weaknesses in model-server scheduling.

I think in the short-term we will remove throughput median from the console output (most other benchmark tools only do mean throughput anyways); it will still be available in the JSON. For longer-term I will keep investigating solutions. One of the things I tried already was a variable threshold that smooths based on a fraction of the token latency. You can see it in the graph above labeled as "variable"; it works fine for output tokens but does not work as well for prompt tokens and requests. Plus over-smoothing is still a concern. Another path I am considering is to plot the PDF just to understand the shape a bit better and then research methods that are more applicable to that shape.

I'll keep working on this in the background but it will be fairly low priority as we have a lot of other work on our plate. Contributions are welcome in the meantime, though anything contributed to this part of the codebase will have to go through more stringent statistical analysis. It would also be good to see if there is a statistics package that does most of what we want and has been audited for correctness. Back when this was implemented we couldn't find one that served our needs well hence why we wrote our own.

[sjmonson]

For reference here are some graphs comparing constant RPS to concurrent. The RPS value used was the measured mean RPS of the concurrent run which is not perfect but at least puts us into a similar workload size.

	concurrent=25	constant=2.34
Measured RPS
Output tokens/s
Input tokens/s
Total tokens/s