feat: support adaptive batching #382
Description
Describe the feature
refer to the paper:
Basically, it will adjust the batch size or wait time according to the load pressure.
This can be helpful when the traffic is not evenly distributed, which is very common. During the low-traffic period, waiting for a large batch size or a long wait time is unnecessary.
Why do you need this feature?
No response
Additional context
No response