[WIP] Investigating why batching streaming is not helping with the streaming overhead

[kouroshHakha]

This PR refactors the LLMServer and vLLMEngine so that streaming batching happens at the LLMServing layer instead of vLLMEngine.

I also noticed that if vLLMEngine does the batching then the LLMServer will unpack the batch and stream the individual items through the remote channel back to the router. This is a problem at high qps where ray's streaming becomes the botteleneck and the whole end to end latency will be impacted by the streaming issue. With proper batching we can mitigate this problem which was not happening before due to the extra unpacking done at the wrong layer. Unpacking should be done at the very last stage when router wants to send back the results to the http proxy.

TODO:

• benchmark with this PR what the impact of batching would be
• See if using pickle / custom serialization helps
• Unittests for batching at llmserver