-
Notifications
You must be signed in to change notification settings - Fork 9
Description
@manoelmarques is benchmarking the launcher and he has good comments to take into account:
1- Parser deprecation
Just to let you know in newer vllms the import for the parser is:
from vllm.utils.argparse_utils import FlexibleArgumentParser
That location has always been valid. The definition has been removed from vllm.utils
2- Requirements
Also, for my dockerfile and in general, the requirements.txt is way too big. SInce I am basing it from a vllm image, I already have everything needed. The only thing I install in the image additionally is uvicorn. I will be testing it and see if it is all that is needed. So far it starts the uvicorn server fine in a pod
Installing the whole requirements.txt using a base vllm image ended up making the base vllm crash with missing C functions and I believe it is because the requirements.txt ended up installing some pytorch version that conflicted with the binaries from the base vllm.