Update optimum_benchmark/backends/vllm/backend.py

IlyasMoutawwakil · web-flow · commit 57d5343b4b30 · 2024-11-25T14:58:06.000+01:00
diff --git a/optimum_benchmark/backends/vllm/backend.py b/optimum_benchmark/backends/vllm/backend.py
@@ -143,7 +143,7 @@ async def single_online_engine_generate(self, prompt: str, request_id: str, kwar
         stream = await self.pretrained_model.add_request(
             inputs=prompt,
             request_id=request_id,
-            params=self.get_sampling_params(),
+            params=self.get_sampling_params(kwargs),
         )
 
         async for _ in stream:

Original file line number	Diff line number	Diff line change
`@@ -143,7 +143,7 @@ async def single_online_engine_generate(self, prompt: str, request_id: str, kwar`
`143`	`143`	`stream = await self.pretrained_model.add_request(`
`144`	`144`	`inputs=prompt,`
`145`	`145`	`request_id=request_id,`
`146`		`- params=self.get_sampling_params(),`
	`146`	`+ params=self.get_sampling_params(kwargs),`
`147`	`147`	`)`
`148`	`148`
`149`	`149`	`async for _ in stream:`