Additional option for alternative model Paths to load different model formats

[John42506176Linux]

Feature request

Additional optional for Docker CLI to load alternative paths of models
port=7997
mid_rerank_model=mixedbread-ai/mxbai-rerank-xsmall-v1
volume=$PWD/data

sudo docker run -it --gpus all
-v $volume:/app/.cache
-p $port:$port
michaelf34/infinity:latest
v2
--batch-size 32
--model-id $mid_rerank_model
--alternative_path onnx/model_quantized.onnx
--port $port

Motivation

I'm currently trying to load the quantized ONNX version of a Reranker model I'm using and I currently can't see an easy solution to doing this with the docker CLI.

Your contribution

Help test or fix few simple bugs.

[michaelfeil]

@John42506176Linux Your request is not realistic. Besides model weights, you also need to load e.g. the config files, tokenizer, and infer the model class from the config.

I would suggest you to pack your custom reranker into a huggingface repo.

I am confused, as for loading onnx models you need to set —engine optimum, which you are not doing.

[John42506176Linux]

Thanks, for the information I didn't know if this was an option so wanted to double-check.

I didn't include the --engine optimum option in the contribution but that was used in testing.

[michaelfeil]

Thanks, I'll move it to discussions.