Closed
Description
Description
This PR honor kPreferredOutputBatchRows config.
#7051
Now there is the constraint that single row output should be into single batch.
But for this case, an input row has a very large nested array+struct, the output batch size is also large.
So we need to respect kPreferredOutputBatchRows strictly.
There is several strategies:
- Split the row only if one row output batch size is more than maxOutputBatchSize.
- Always split the last row to match the output batch size .
I would prefer the second way, it can lead to accurate batch size.
We could add a benchmark to test the performance if we always split the end row.