feat: [velox][iceberg] Add DWRF file format support for Iceberg data sink#16875
Open
apurva-meta wants to merge 7 commits intofacebookincubator:mainfrom
Open
feat: [velox][iceberg] Add DWRF file format support for Iceberg data sink#16875apurva-meta wants to merge 7 commits intofacebookincubator:mainfrom
apurva-meta wants to merge 7 commits intofacebookincubator:mainfrom
Conversation
Summary: - Add explicit equality deletes NYI branch in prepareSplit() - Improve VELOX_NYI error messages with descriptive content type info - Fix FILE handle leaks in IcebergReadTest by extracting getTestFileSize() helper - Minor doc comment formatting improvements in IcebergSplitReader.h Differential Revision: D97530140
Summary: Add deletion vector (DV) reader to the Velox Iceberg connector, enabling Iceberg V3 spec support for row-level deletes. Iceberg V3 replaces positional delete files with deletion vectors — compact roaring bitmaps stored as blobs inside Puffin files. Compared to V2 positional delete files, DVs are more compact and avoid sorted merge of multiple delete files at read time. Changes: - IcebergDeleteFile.h: Add FileContent::kDeletionVector enum value - DeletionVectorReader.h/cpp: New reader that loads a Puffin blob, deserializes the roaring bitmap portable format (array, bitset, and run containers), and sets bits in the deleteBitmap for the current batch range. No CRoaring dependency — self-contained parser. - IcebergSplitReader.h: Add deletionVectorReaders_ member, include header - IcebergSplitReader.cpp: Route kDeletionVector in prepareSplit(), apply DVs alongside positional deletes in next() - BUCK: Add DeletionVectorReader library target Differential Revision: D97530142
Summary: Implements Iceberg equality delete support for the Velox Iceberg connector. Equality delete files contain rows with values for one or more columns (identified by equalityFieldIds). A base data row is deleted if its values match ALL specified columns of ANY row in the delete file. The implementation: - Adds EqualityDeleteFileReader that eagerly reads the entire delete file and builds an in-memory hash multimap of delete key tuples during construction. - Wires EqualityDeleteFileReader into IcebergSplitReader::prepareSplit() to resolve equalityFieldIds to column names/types from the table schema, and into IcebergSplitReader::next() to apply post-read equality delete filtering with row compaction. - Handles lazy vectors from file readers via loadedVector() before accessing values for hashing and comparison. - Supports BOOLEAN, TINYINT, SMALLINT, INTEGER, BIGINT, REAL, DOUBLE, VARCHAR, VARBINARY, and TIMESTAMP column types. Differential Revision: D97530141
Summary: Implements Iceberg V2+ sequence number conflict resolution for equality delete files. Per the Iceberg spec, an equality delete file should only be applied to data files whose data sequence number is strictly less than the delete file's data sequence number. This prevents concurrent write conflicts where a delete file written concurrently with a data file could incorrectly delete rows that were not intended to be deleted. Changes: - Added `dataSequenceNumber` field to `IcebergDeleteFile` struct with default value 0 (unassigned/legacy V1). When 0, sequence number filtering is disabled for backward compatibility. - Added `dataSequenceNumber` field to `HiveIcebergSplit` to carry the base data file's sequence number. - Updated `IcebergSplitReader::prepareSplit()` to skip equality delete files when `deleteFile.dataSequenceNumber <= split.dataSequenceNumber` (unless either is 0, which disables the check). - Updated test constructor of `HiveIcebergSplit` to accept the new `dataSequenceNumber` parameter. Differential Revision: D97530136
…letion vectors Summary: Extend the sequence number conflict resolution logic to positional deletes and deletion vectors. Per the Iceberg spec: - Positional deletes and DVs skip when deleteFileSeqNum < dataFileSeqNum (strictly less than, unlike equality deletes which use <=) - Sequence number 0 (V1 legacy) never skips Changes: - IcebergSplitReader: Apply sequence number filtering for positional deletes and deletion vectors before passing to readers - IcebergReadFile: Store dataSequenceNumber for conflict resolution - Tests: Add sequence number conflict resolution tests for positional deletes and DVs Differential Revision: D97530139
Summary: - Add DeletionVectorWriter.cpp/h implementing DV file writing using RoaringBitmapArray - Support both position-based (positional deletes) and DV-based delete file writing - Add DeletionVectorWriterTest with comprehensive unit tests - Update BUCK and CMakeLists.txt build targets Differential Revision: D97530137
…sink Summary: - Add DWRF file format support in IcebergDataSink for read and write paths - Update BUCK build targets for DWRF dependencies - Add IcebergDwrfInsertTest with comprehensive insert/read tests for DWRF format - Update CMakeLists.txt for new test targets Differential Revision: D97530138
✅ Deploy Preview for meta-velox canceled.
|
meta-cla
bot
added
the
CLA Signed
|
@apurva-meta has exported this pull request. If you are a Meta employee, you can view the originating Diff in D97530138. |
PingLiuPing
reviewed
Mar 21, 2026
Collaborator
PingLiuPing
left a comment
PingLiuPing
left a comment
There was a problem hiding this comment.
Hi @apurva-meta, thanks for the code.
I noticed that several recently opened PRs seem to overlap. Could you please open a GitHub issue outlining your overall plan and how these changes fit together? If the work is still in progress, it would be great to mark the PRs as drafts for now.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Differential Revision: D97530138