Multi Process Scheduler Implementation, Benchmarker, and Report Generation Refactor #96

markurtz · 2025-03-10T15:28:00Z

Full refactor of GuideLLM enabling better overall performance to ensure minimal overhead for benchmarking with a new multiprocess and threaded scheduler along with significant updates to the output formats enabling better analysis, visibility, and clarity.

Fixes:

…http/2 support. Additionally, add in support for multi modal requests (needs further enablement in the rest of the system for future TODO). Still needs testing and test fixes

…/ in place

…urther integration / refactors.

…ric with how scheduler uses them

…equired changes, migrations, and fixes

sjmonson

Not a complete review; still working on root causing some other issues I found.

src/guidellm/utils/random.py

src/guidellm/__init__.py

src/guidellm/scheduler/scheduler.py

…ed on testing

Mypy gets unhappy when we instantiate a generic class without declaring its type (aka infer from the inital value). As a workaround just declare our default type as Any.

f"{"test"}" is valid in 3.12 but not in older pythons

In python < 3.12 importlib.resources.files will fail if the module contains no python.

…eedback

sjmonson

~~LGTM 📦➡️🚚~~ Actually one minor change needed

sjmonson

~~Now LGTM~~ 🤣 or not. Since python 3.9 adds support for using list and dict as types this has made ruff unhappy with using the typing constructs of List and Dict. Reverting my change for now we can revisit in separate PR.

Since python 3.9 adds support for using list and dict as types this has made ruff unhappy with using the typing constructs of List and Dict. Reverting my change for now; we can revisit in separate PR. This reverts commit d0d31d3.

markurtz changed the base branch from main to openai_backend_fixes March 10, 2025 15:28

markurtz self-assigned this Mar 10, 2025

Base automatically changed from openai_backend_fixes to main March 12, 2025 15:42

rgreenberg1 added this to the GuideLLM v0.2.0 - CI/CD Finalization, Documentation Expansion, and Backend Support milestone Mar 13, 2025

rgreenberg1 added the load-request load-request workstream label Mar 13, 2025

rgreenberg1 added this to GuideLLM Kanban Board Mar 13, 2025

rgreenberg1 mentioned this pull request Mar 13, 2025

Concurrent load generation option is implemented #71

Closed

rgreenberg1 added the priority-high label Mar 13, 2025

anmarques and others added 9 commits March 15, 2025 14:53

Adds aiohttp backend

1e21612

quality fixes

ce0c3c5

Quality fixes

8cb9876

Rework for OpenAI backend to use native http requests with httpx and …

039900a

…http/2 support. Additionally, add in support for multi modal requests (needs further enablement in the rest of the system for future TODO). Still needs testing and test fixes

Finalize implementation, fix bugs, and ensure unit tests are passing …

2b82bd5

…/ in place

Initial state for Scheduling system rework. Still needs testing and f…

a01db7d

…urther integration / refactors.

Enable temp testing script to work, refactor strategy to be more gene…

85dee09

…ric with how scheduler uses them

Polishing for scheduler

fb3fdd7

styling fixes

12835e0

markurtz force-pushed the perf_fixes branch from a646d2a to 12835e0 Compare March 15, 2025 15:06

markurtz added 5 commits March 17, 2025 05:06

initial groundwork for new benchmark package and classes along with r…

65869d7

…equired changes, migrations, and fixes

finalize benchmark model objects

53e0943

Latest state with datasets and benchmarker created

faacb63

latest working state with data loaders and benchmarking API

34f851c

Working clis and entrypoints

cfdc2ed

sjmonson requested changes Apr 8, 2025

View reviewed changes

src/guidellm/utils/random.py Show resolved Hide resolved

src/guidellm/__init__.py Outdated Show resolved Hide resolved

src/guidellm/scheduler/scheduler.py Outdated Show resolved Hide resolved

Fixes for bugs surfaced from testing and enhancements to features bas…

d4f8c1a

…ed on testing

sjmonson and others added 13 commits April 10, 2025 23:19

Define sampling sizes and counts as StatusBreakdowns

31adea4

Set a default type for SuccessfulT

f8c5e7a

Mypy gets unhappy when we instantiate a generic class without declaring its type (aka infer from the inital value). As a workaround just declare our default type as Any.

Fix case chnage on requests_per_second

93f0fd1

Plumb output changes though progress and summary

00a210d

Pluralization is hard

cf160b6

Fix changes after rebase

2bedc6d

Fix/ignore linting errors due to line length changes

331978b

Fix double quotes inside a double qoute f-string

c449bde

f"{"test"}" is valid in 3.12 but not in older pythons

importlib.resources.files requires valid module

4cd904d

In python < 3.12 importlib.resources.files will fail if the module contains no python.

Fix for restructuring of output and general simplification based on f…

48098fc

…eedback

Fix quality, unit, integration, and e2e tests

0263361

Just kidding, let's try that again and hopefully fix quality and tests

b164b4b

Trying one more time for quality

b411db3

markurtz marked this pull request as ready for review April 11, 2025 16:42

markurtz changed the title ~~General Multi Process Scheduler Implementation and Report Generation Updates~~ Multi Process Scheduler Implementation, Benchmarker, and Report Generation Refactor Apr 11, 2025

sjmonson previously approved these changes Apr 11, 2025

View reviewed changes

Bump min python to 3.9

d0d31d3

sjmonson dismissed their stale review via d0d31d3 April 11, 2025 19:59

sjmonson previously approved these changes Apr 11, 2025

View reviewed changes

Revert "Bump min python to 3.9"

70aa669

Since python 3.9 adds support for using list and dict as types this has made ruff unhappy with using the typing constructs of List and Dict. Reverting my change for now; we can revisit in separate PR. This reverts commit d0d31d3.

sjmonson dismissed their stale review via 70aa669 April 11, 2025 20:06

sjmonson approved these changes Apr 11, 2025

View reviewed changes

markurtz merged commit 09655a4 into main Apr 11, 2025
9 checks passed

markurtz deleted the perf_fixes branch April 11, 2025 20:56

github-project-automation bot moved this to Done in GuideLLM Kanban Board Apr 11, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi Process Scheduler Implementation, Benchmarker, and Report Generation Refactor #96

Multi Process Scheduler Implementation, Benchmarker, and Report Generation Refactor #96

markurtz commented Mar 10, 2025 •

edited

Loading

sjmonson left a comment

sjmonson left a comment •

edited

Loading

sjmonson left a comment •

edited

Loading

Multi Process Scheduler Implementation, Benchmarker, and Report Generation Refactor #96

Multi Process Scheduler Implementation, Benchmarker, and Report Generation Refactor #96

Conversation

markurtz commented Mar 10, 2025 • edited Loading

sjmonson left a comment

Choose a reason for hiding this comment

sjmonson left a comment • edited Loading

Choose a reason for hiding this comment

sjmonson left a comment • edited Loading

Choose a reason for hiding this comment

markurtz commented Mar 10, 2025 •

edited

Loading

sjmonson left a comment •

edited

Loading

sjmonson left a comment •

edited

Loading