Using Data parallel to scale to multiple GPUs for LLama-3.1-8B

[naveenmiriyaluredhat]

Investigate using DP with the harness to scale to multiple replicas .

[naveenmiriyaluredhat]

1. Offline scenario with DP=2 and LLM API hangs . Suspecting issues how harness handles multi-process spawn
   The model loads. Samples are sent and then its a hang.
2. What happens with DP when one prompt with multiple queries is sent at once ? Code ? How are the prompts distributed and how is the load-balancing done ? Any existing performance tuning option for DP load balancing ?
3. An OpenAPI server with DP=2 does not hang
4. DP=1 gave 5249 toks/sec and DP=2 9096 toks/sec. Scaling factor ~1.7 . Almost 30% loss with scaling
5. LLM-API - 5777 toks/sec and with API Server - 5249 toks/sec . 10% loss in perf ?? Investigate reasons and causes ?
6. Investigate with harness load balancing multiple endpoints ?
7. When running with DP=2, does the vLLM metrics endpoint provide performance information for each individual engine (replica)?