compared to java ORC, Velox has a more significant read amplification issue

[zhaiyuyong0412-dot]

The Java ORC logic first filters out the position of each row group based on the pushdown conditions, and then performs a merge
However, when Velox reads from Orc, the row group skip function becomes ineffective when multiple columns are merged

[kagamiori]

Hi @zhaiyuyong0412-dot, thank you for initiating this discussion. Could you add some code pointers to help us better identify where the problem is?

cc @Yuhta

[zhaiyuyong0412-dot]

We need to generate an ORC file tailored to our business scenario.

[Yuhta]

I am not sure what the issue is, we have row group level skipping and merging of the conditions, we also have row level skipping that Apache ORC reader does not have.

[zhaiyuyong0412-dot]

Based on my understanding of the DirectBufferedInput::groupRequests function, it merges a subset of streams into a DirectCoalescedLoad.
During the actual read process, when reading one of the streams that belongs to a DirectCoalescedLoad, it reads the data for the entire coalesced load at once (which includes the complete data of all merged streams). In doing so, it seems to bypass the effect of the previously passed seekToRowGroup. Is this behavior expected?
Additionally, in the Java ORC implementation, separate reads are coalesced to reduce the number of IO operations. Does Velox have a similar design for IO coalescing?