Scan Delete Support Part 4: Delete File Loading; Skeleton for Processing #982
Conversation
sdd
force-pushed
the
feat/delete-fila-manager-loading
branch
5 times, most recently
from
edb1d27 to
8e90bdd
Compare
sdd
force-pushed
the
feat/delete-fila-manager-loading
branch
4 times, most recently
from
ec8e7c1 to
06f0df5
Compare
| struct DeleteFileManagerState { | ||
| // delete vectors and positional deletes get merged when loaded into a single delete vector | ||
| // per data file | ||
| delete_vectors: HashMap<String, RoaringTreemap>, |
There was a problem hiding this comment.
I think a function should be included for enabling deletion vectors for the future when a property is added.
There was a problem hiding this comment.
Sorry @jonathanc-n, I'm a bit confused as to what you mean, could you explain?
sdd
force-pushed
the
feat/delete-fila-manager-loading
branch
6 times, most recently
from
5530bc3 to
e997fc6
Compare
|
@liurenjie1024, @Xuanwo, @Fokko - this is ready for re-review, if you could take a look that would be great! |
sdd
force-pushed
the
feat/delete-fila-manager-loading
branch
from
e997fc6 to
056e73f
Compare
There was a problem hiding this comment.
Thanks @sdd for this pr. There are some missing points in current design. Also I would suggest not putting too much in DeleteFilterManager. I suppose DeleterFilterManager acting more like a delete loader, which manages the io and caching of record batch. The actual filtering part, could delegate to DeleteFilter, WDYT? I think a good reference implementation is java's DeleteFilter, see https://github.com/apache/iceberg/blob/af8e3f5a40f4f36bbe1d868146749e2341471586/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L50
| /// Parses record batch streams from individual equality delete files | ||
| /// | ||
| /// Returns an unbound Predicate for each batch stream | ||
| async fn parse_equality_deletes_record_batch_stream( |
There was a problem hiding this comment.
Transforming eq deletes to predicate maybe in efficient for arrow. In theory, eq delete filter is just an null safe anti join, maybe calling some arrow kernel would be more efficient.
There was a problem hiding this comment.
Seeing as this is around efficiency rather than correctness, would you be ok with deferring a more efficient implementation to a follow-up PR? I'd prefer to have some correct support for delete files sooner and then refine it to perform better afterwards, if that is ok with you.
|
Thanks for the review @liurenjie1024 - much appreciated. Will come back with a revised design. |
sdd
force-pushed
the
feat/delete-fila-manager-loading
branch
2 times, most recently
from
bd33aa5 to
39a26ab
Compare
sdd
force-pushed
the
feat/delete-fila-manager-loading
branch
2 times, most recently
from
58b0f07 to
5739a46
Compare
…DeleteVector::intersect_assign pub(crate)
sdd
force-pushed
the
feat/delete-fila-manager-loading
branch
from
5739a46 to
52cf8b9
Compare
|
Back to you @liurenjie1024 - I've made the changes around missing functionality. Still the open question of if you are ok to defer the structural / performance changes to a follow-up so that we can make more incremental progress rather than adding yet more changes into this PR. |
Extends the
DeleteFileManagerintroduced in #950 To include loading of delete files, storage and retrieval of parsed delete files from shared state, and the outline for how parsing will connect up to this new work.Issue: #630