Skip to content

feat(haproxy-load-balancer): add haproxy load balancer #241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 165 commits into
base: main
Choose a base branch
from
Open

feat(haproxy-load-balancer): add haproxy load balancer #241

wants to merge 165 commits into from

Conversation

@AWarno
Copy link
Contributor

@AWarno AWarno commented Sep 29, 2025 •
edited
Loading

Enabling Multi-Instance Deployment with HAProxy

Why HAProxy

HAProxy is a lightweight, reliable, and widely used load balancer. It generalizes well to all server types. Using an external load balancer is officially recommended in the vLLM documentation (see vLLM Data Parallel Deployment); the documentation provides an example using NGINX, but HAProxy should work similarly.

Alternative Solutions

  • Ray
    This is useful for multi-node deployments when a model is too large for a single node. It can also be used for multi-instance setups, but it requires knowing how to launch and manage each server type individually (vLLM, SGLang may have different CLI arguments for this). It does not generalize as well as using an external load balancer. However, we may want to provide an example of how to use it for multi-node large model deployment.

  • LiteLLM
    Offers backend orchestration but is generally overkill for simple load balancing. The project evolves quickly, which may affect stability.

  • NGINX
    Very similar to HAProxy for this use case and officially recommended in the vLLM documentation:
    vLLM Data Parallel Deployment
    HAProxy, however, is slightly simpler/nicer to use in practice (based on my experience).

Literature

TODO

  • Run on longer tasks to validate stability and performance. (I have checked ifeval so far)
  • Check if the HAProxy template is correctly included in the pip wheel (consider renaming it)
  • Documentation
  • dataclass in types fix!!!!

Next Steps

  • Add a multi-node deployment example using Ray server. This will likely just require creating one example configuration file under examples/.

@AWarno AWarno requested review from a team and agronskiy as code owners September 29, 2025 11:45
@copy-pr-bot
Copy link

copy-pr-bot bot commented Sep 29, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

ko3n1g and others added 24 commits September 29, 2025 13:56
@ko3n1g @AWarno
beep boop 🤖: Bumping nemo_evaluator_launcher to v0.1.4
1326d00 
Signed-off-by: Oliver Koenig <[email protected]>
Signed-off-by: Anna Warno <[email protected]>
@ko3n1g @AWarno
beep boop 🤖: Bumping nemo_evaluator to v0.1.4
3a846a9 
Signed-off-by: Oliver Koenig <[email protected]>
Signed-off-by: Anna Warno <[email protected]>
@agronskiy @AWarno
chore(ci/release): enable cron run (#216)
098f3fb 
Signed-off-by: Anna Warno <[email protected]>
@wprazuch @AWarno
(feat) Configure request method for progress tracking requests (#213)
da80190 
Signed-off-by: Wojciech Prazuch <[email protected]>
Signed-off-by: Anna Warno <[email protected]>
@ko3n1g @AWarno
beep boop 🤖: Bumping nemo_evaluator_launcher to v0.1.5
f3e36e7 
Signed-off-by: Oliver Koenig <[email protected]>
Signed-off-by: Anna Warno <[email protected]>
@ko3n1g @AWarno
beep boop 🤖: Bumping nemo_evaluator to v0.1.5
7f77d93 
Signed-off-by: Oliver Koenig <[email protected]>
Signed-off-by: Anna Warno <[email protected]>
@AWarno
Awarno/reasoning tokens (#211)
6ae471c 
1. Add total stats.
2. Add reasoning token stats (if provided). -
https://platform.openai.com/docs/guides/reasoning or "reasoning_tokens"
in usage, (completion_tokens_details, output_tokens_details)
3. Make stats cache-resistant — do not include stats if the response is
from cache.

---------

Signed-off-by: Anna Warno <[email protected]>
@AWarno
Update overview.md (#210)
6e81164 
checkbox added

Signed-off-by: AWarno <[email protected]>
Signed-off-by: Anna Warno <[email protected]>
@AWarno
feat(multi-instance): haproxy
75aeeb6 
Signed-off-by: Anna Warno <[email protected]>
@AWarno
fix(health-url): health url fixed
86a557d 
Signed-off-by: Anna Warno <[email protected]>
@AWarno
fix(conflict): fix conflict
0681135 
Signed-off-by: Anna Warno <[email protected]>
@ko3n1g @AWarno
beep boop 🤖: Bumping nemo_evaluator to v0.1.4
710ca95 
Signed-off-by: Oliver Koenig <[email protected]>
@ko3n1g @AWarno
beep boop 🤖: Bumping nemo_evaluator_launcher to v0.1.5
fc772a2 
Signed-off-by: Oliver Koenig <[email protected]>
Signed-off-by: Anna Warno <[email protected]>
@ko3n1g @AWarno
beep boop 🤖: Bumping nemo_evaluator to v0.1.5
a5bee56 
Signed-off-by: Oliver Koenig <[email protected]>
Signed-off-by: Anna Warno <[email protected]>
@piojanu @AWarno
fix(executors): migrate to eval-factory cmd (#229)
4884615 
It unblocks us to use new Eval Factory containers in the launcher — they
don't have `nv-eval`/`nv_eval` alias anymore.

Signed-off-by: Piotr Januszewski <[email protected]>
Signed-off-by: Anna Warno <[email protected]>
@wprazuch @AWarno
(chore) Update container versions (#230)
d6cf567 
Signed-off-by: Wojciech Prazuch <[email protected]>
Signed-off-by: Anna Warno <[email protected]>
@lbliii @jgerh @AWarno
Llane/home pg style edits (#215)
5bf95ad 
This is a very basic migration of the readme content + adding a minimal
toctree to the home index page so that the sphinx site produces a
sidebar. The sidebar will mature and break out in the future into
sections such as About, Get Started, etc.

We will also add more sections/cards to this page after all other basic
edits have been checked in, so it won't be a direct copy of the README,
instead it will become a proper docs site home page.

---------

Signed-off-by: Lawrence Lane <[email protected]>
Signed-off-by: L.B. <[email protected]>
Co-authored-by: jgerh <[email protected]>
Signed-off-by: Anna Warno <[email protected]>
@wprazuch @AWarno
(chore) Switch to referring to latest in the docs (#231)
de17a9e 
Signed-off-by: Wojciech Prazuch <[email protected]>
Signed-off-by: Anna Warno <[email protected]>
@AWarno @ko3n1g
@agronskiy @wprazuch
post qa docs update (#225)
693392c 
Docs update

---------

Signed-off-by: Anna Warno <[email protected]>
Signed-off-by: Oliver Koenig <[email protected]>
Signed-off-by: Wojciech Prazuch <[email protected]>
Signed-off-by: AWarno <[email protected]>
Co-authored-by: Oliver Koenig <[email protected]>
Co-authored-by: Alexey Gronskiy <[email protected]>
Co-authored-by: Wojciech Prazuch <[email protected]>
Signed-off-by: Anna Warno <[email protected]>
@wprazuch @AWarno
(chore) Revert switch to latest (#232)
6aa2f8c 
Signed-off-by: Wojciech Prazuch <[email protected]>
Signed-off-by: Anna Warno <[email protected]>
@ko3n1g @AWarno
beep boop 🤖: Bumping nemo_evaluator to v0.1.6
4375a60 
Signed-off-by: Oliver Koenig <[email protected]>
Signed-off-by: Anna Warno <[email protected]>
@ko3n1g @AWarno
beep boop 🤖: Bumping nemo_evaluator_launcher to v0.1.6
a1fb564 
Signed-off-by: Oliver Koenig <[email protected]>
Signed-off-by: Anna Warno <[email protected]>
@ko3n1g @AWarno
ci(fix): Dependabot (#236)
c78bb73 
Signed-off-by: oliver könig <[email protected]>
Signed-off-by: Anna Warno <[email protected]>
@marta-sd @AWarno
Add changes from PR 215 to the README (#239)
1452fce 
Signed-off-by: Marta Stepniewska-Dziubinska <[email protected]>
Signed-off-by: Anna Warno <[email protected]>
@AWarno
Copy link
Contributor Author

AWarno commented Oct 23, 2025

/ok to test b14f3d7

@copy-pr-bot
Copy link

copy-pr-bot bot commented Oct 23, 2025

/ok to test b14f3d7

@AWarno, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

@AWarno
Copy link
Contributor Author

AWarno commented Oct 23, 2025

/ok to test e2fb5ce

@github-actions github-actions bot added the tests label Oct 23, 2025
fgalko-oss
fgalko-oss previously approved these changes Oct 27, 2025
View reviewed changes
@AWarno
Copy link
Contributor Author

AWarno commented Oct 27, 2025

/ok to test 10d1c02

@AWarno
Copy link
Contributor Author

AWarno commented Oct 27, 2025

/ok to test f74f6c4

@fgalko-oss fgalko-oss self-requested a review October 28, 2025 04:49
fgalko-oss
fgalko-oss previously approved these changes Oct 28, 2025
View reviewed changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fgalko-oss fgalko-oss fgalko-oss approved these changes

@agronskiy agronskiy Awaiting requested review from agronskiy

Assignees

No one assigned

Labels

nemo-evaluator-launcher tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.