Commit 1d579f6
authored
Pass mp context to strategy (#651)
## Summary
Fixes spawn and forkserver multi-process contexts.
## Details
I was hoping that after #647 we could switch to `forkserver` by default.
However it turns out that `forkserver` and `spawn` will import the
calling processes entrypoint (E.g. `__main__.py`) so we run into the
same blocker as #641. However, I was able to confirm that striping every
heavy import out of `__main__.py` solves the issue. So we should be good
to switch in v0.7.0.
On my machine there is about a ~10s overhead for `forkserver` and
slightly more for `spawn`, which is not the worst for a default.
However, the overhead may be more on other systems:
### `time guidellm benchmark run --profile poisson --rate 5 --data
prompt_tokens=128,output_tokens=128 --max-seconds 30 --outputs json`
| Context | real | user | sys |
| ---------- | --------- | --------- | -------- |
| Fork | 0m37.874s | 0m17.356s | 0m1.883s |
| Forkserver | 0m47.344s | 0m14.862s | 0m0.860s |
| Spawn | 0m49.515s | 1m51.230s | 0m8.915s |
### `time guidellm benchmark run --profile concurrent --rate 400 --data
prompt_tokens=128,output_tokens=128 --max-seconds 30 --outputs json`
| Context | real | user | sys |
| ---------- | --------- | --------- | --------- |
| Fork | 0m39.324s | 0m37.602s | 0m5.623s |
| Forkserver | 0m49.609s | 0m19.710s | 0m1.311s |
| Spawn | 0m50.399s | 2m9.724s | 0m11.374s |
### `time guidellm benchmark run --profile concurrent --rate 400 --data
prompt_tokens=128,output_tokens=128 --max-seconds 120 --outputs json`
| Context | real | user | sys |
| ---------- | --------- | --------- | --------- |
| Fork | 2m15.309s | 1m42.911s | 0m15.957s |
| Forkserver | 2m25.964s | 0m38.891s | 0m2.802s |
| Spawn | 2m27.454s | 3m24.325s | 0m22.531s |
## Test Plan
Set `GUIDELLM__MP_CONTEXT_TYPE=forkserver` and confirm benchmarks run.
---
- [x] "I certify that all code in this PR is my own, except as noted
below."
## Use of AI
- [x] Includes AI-assisted code completion
- [ ] Includes code generated by an AI application
- [ ] Includes AI-generated tests (NOTE: AI written tests should have a
docstring that includes `## WRITTEN BY AI ##`)File tree
3 files changed
+36
-14
lines changed- src/guidellm/scheduler
- tests/unit/scheduler
3 files changed
+36
-14
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| |||
103 | 104 | | |
104 | 105 | | |
105 | 106 | | |
106 | | - | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
107 | 111 | | |
108 | 112 | | |
109 | 113 | | |
| |||
117 | 121 | | |
118 | 122 | | |
119 | 123 | | |
120 | | - | |
121 | | - | |
122 | | - | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
123 | 127 | | |
124 | 128 | | |
125 | 129 | | |
| |||
593 | 597 | | |
594 | 598 | | |
595 | 599 | | |
596 | | - | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
597 | 606 | | |
598 | 607 | | |
599 | 608 | | |
| |||
603 | 612 | | |
604 | 613 | | |
605 | 614 | | |
606 | | - | |
| 615 | + | |
607 | 616 | | |
608 | 617 | | |
609 | | - | |
| 618 | + | |
610 | 619 | | |
611 | 620 | | |
612 | 621 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
221 | 221 | | |
222 | 222 | | |
223 | 223 | | |
224 | | - | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
225 | 227 | | |
226 | 228 | | |
227 | 229 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
5 | 6 | | |
6 | 7 | | |
7 | 8 | | |
| |||
502 | 503 | | |
503 | 504 | | |
504 | 505 | | |
505 | | - | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
506 | 509 | | |
507 | 510 | | |
508 | 511 | | |
| |||
525 | 528 | | |
526 | 529 | | |
527 | 530 | | |
528 | | - | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
529 | 534 | | |
530 | 535 | | |
531 | 536 | | |
| |||
574 | 579 | | |
575 | 580 | | |
576 | 581 | | |
577 | | - | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
578 | 585 | | |
579 | 586 | | |
580 | 587 | | |
| |||
584 | 591 | | |
585 | 592 | | |
586 | 593 | | |
587 | | - | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
588 | 597 | | |
589 | 598 | | |
590 | 599 | | |
| |||
613 | 622 | | |
614 | 623 | | |
615 | 624 | | |
616 | | - | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
617 | 628 | | |
618 | 629 | | |
619 | 630 | | |
| |||
0 commit comments