How to change the service's batch-size

[TOMATODA]

I already deployed the model service in the container successfully using the default batch-size , I want to increase the batch-size，what should I do to accomplish it without launching the container once again?

deploy command as follows:
/app/.venv/bin/python /app/.venv/bin/infinity_emb --model-name-or-path /opt/llm/input/pretrain/BAAI--bge-reranker-v2-m3 --port 8002 --device cuda --dtype float16

[kimson99]

You can you --batch-size to change it on startup, checkout infinity_emb v2 --help for more options. I don't think you can change it while running.

[michaelfeil]

This is correct! --batch-size sets the maximum allowed batch size. Set it to 32 or 64. You will likely see marginal returns beyond.
Currently, resizing the batch size after warmup could be done, but maybe would lead to re-compilation if using (tensorrt, torch compile)

[TOMATODA]

thank you both !