Adding support for CSV files#46
Conversation
| * Regular seek. Called once per offset (block). | ||
| */ | ||
| override def seek(offset: Long): Unit = { | ||
| // @TODO - need to better optimize. not to re-init on every seek |
There was a problem hiding this comment.
So I guess this is only for random fetches, right? The performance will be impractical for batch...
There was a problem hiding this comment.
but than again, why not just save the reader open? am I missing anything?
There was a problem hiding this comment.
currently, the requested use-case is only fetches. but sure I would like to solve for both. started just with something that works.
In CSV we have only offset, sub-offset is almost only zero.
let's say we want to read every other row, then we need to understand when it will be better just to skip a row compared to when will we want to "re-init" from a new offset.
There was a problem hiding this comment.
ok I got it... In the parquet implementation we have basically the same thing, right? I mean just one offset, when the row group is 128MB or more
2 participants
Summary
Adding support for indexing CSV files.
How was it tested?
added unit tests