Skip to content

Commit adfa108

Browse files
authored
Allow unlimited connections per-worker (#488)
## Summary <!-- Include a short paragraph of the changes introduced in this PR. If this PR requires additional context or rationale, explain why the changes are necessary. --> By default each httpx client supports a maximum of 100 connections ([ref](https://www.python-httpx.org/advanced/resource-limits/)). We want this uncapped as connection count is maintained by a semaphore. ## Test Plan <!-- List the steps needed to test this PR. --> - See #487 ## Related Issues <!-- Link any relevant issues that this PR addresses. --> - Resolves #487 --- - [x] "I certify that all code in this PR is my own, except as noted below." ## Use of AI - [ ] Includes AI-assisted code completion - [ ] Includes code generated by an AI application - [ ] Includes AI-generated tests (NOTE: AI written tests should have a docstring that includes `## WRITTEN BY AI ##`)
2 parents a98a644 + abbc62c commit adfa108

File tree

1 file changed

+6
-0
lines changed

1 file changed

+6
-0
lines changed

src/guidellm/backends/openai.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,12 @@ async def process_startup(self):
134134
timeout=self.timeout,
135135
follow_redirects=self.follow_redirects,
136136
verify=self.verify,
137+
# Allow unlimited connections
138+
limits=httpx.Limits(
139+
max_connections=None,
140+
max_keepalive_connections=None,
141+
keepalive_expiry=5.0, # default
142+
),
137143
)
138144
self._in_process = True
139145

0 commit comments

Comments
 (0)