[FEAT] Add replay from trace strategy

[VincentG1234]

Summary

• Add a new replay benchmarking strategy that reproduces real-world request patterns from trace log files (.jsonl)
• Enable time-based request rate replay with precise timestamp scheduling
• Support synthetic prompt generation that matches token counts from trace files
• use max_requests and max_seconds cli options to limit the number of requests processed from a trace

Motivation

This change addresses issue #597 by enabling users to benchmark their vLLM servers using real production traces. Instead of synthetic load patterns, users can now replay exact request arrival times and token distributions from their actual workloads for more realistic performance testing.

Changes

• Add TraceReplayStrategy scheduler strategy for timestamp-based request dispatching
• Add ReplayProfile class for configuring trace-based benchmarking parameters
• Add TraceSyntheticDatasetDeserializer to generate prompts matching trace input/output lengths
• Add TraceReader utility for reading .jsonl trace files with timestamp, input_length, output_length fields
• Update Entrypoint to handle replay profile and dataset configuration
• use max_requests and max_seconds truncation support to limit trace replay length

Testing

• pytest tests/unit/scheduler/test_trace_replay.py (pass)

• pytest tests/unit/benchmark/test_replay_profile.py (pass)

• pytest tests/unit/data/deserializers/test_trace_synthetic.py (pass)

• Added tests: scheduling accuracy, boundary conditions, malformed trace handling, empty trace cases, max_requests truncation

• test it in practice quickly with NB COLAB

Next Steps (this PR)

1. Apply reviewer feedback
2. Add E2E tests verifying end-to-end trace replay flow ✅
3. Add integrations tests (if needed)
4. Add CLI usage examples in PR description and docs ✅

Out of Scope (future PRs or not)

• Mooncake trace format support (token-level traces)
• Helper utilities for timestamp format conversions (Unix epoch, ISO8601, relative timestamps)
• Support for request payload traces (not just token counts)
• Trace file validation and schema verification tools
• Performance optimizations for large trace files (streaming, chunked processing)
• Metrics export formatted for trace analysis comparison
• Support for trace file compression formats (.gz, .bz2)

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @VincentG1234.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[hgsmn]

It will be great to get an example of "How to get the JSONL" because i don't find solutions in litellm for example.

[VincentG1234]

Yeah that’s true, most frameworks won’t produce this exact JSONL directly.

That’s kind of intentional. The idea here is to define a minimal, framework-agnostic canonical replay format, not something tied to a specific tracing stack.

In practice, the required fields already exist almost everywhere (timestamp, input token count, output token count), just under slightly different names, so a small mapping step is usually enough.

I agree it’s not the best UX on its own, but it felt like the right minimal base for the feature. Then we can iterate on top of it with helpers / converters for common sources like LiteLLM or Langfuse. And we can extend it later (e.g. optional prompt field, multiple timestamp formats, richer metadata) without breaking the core idea.

But happy to adjust the direction if maintainers prefer something more opinionated or integrated from the start.