[INFO] How to scale ?

[tsensei]

Title should say it. But how do we actually scale for more throughput? For example let's say I use just one model for embedding. It's working fine but I need more throughput, do I deploy same model multiple times like embedding-model-1, embedding-model-2 and such? Because from my understanding, there is an internal queue for incoming requests and requests and continuously batched and sent to the model.

Thanks

[michaelfeil]

@tsensei One replica should utlize one GPU or a CPU of one host fully.

Assuming you have 1 GPU on your host, there is nothing you can do - you need to SCALE horizontally (kubernetes style horizontal pod autoscaler).
There are also plenty of examples on e.g. modal & runpod have built in autoscaling.
Just use there solutions if you dont have a kubernetes cluster.