Add more columns to Iceberg $files system table by oskar-szwajkowski · Pull Request #29044 · trinodb/trino

oskar-szwajkowski · 2026-04-08T18:19:36Z

Description

This PR enhances Iceberg's connector $files system table by adding additional columns like:
file_sequence_number, data_sequence_number, referenced_data_file, pos, manifest_location, first_row_id, content_offset, and content_size_in_bytes

Those are mostly useful for an insights of how does delete file relate to data files

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(X) Release notes are required, with the following suggested text:

* Extend $files system table in Iceberg connector with additional columns

docs/src/main/sphinx/connector/iceberg.md

findinpath · 2026-04-09T08:55:01Z

...n/trino-iceberg/src/main/java/io/trino/plugin/iceberg/system/files/FilesTablePageSource.java

            // readable_metrics
            writeValueOrNull(pageBuilder, READABLE_METRICS_COLUMN_NAME, () -> metadataSchema.findField(MetricsUtil.READABLE_METRICS),
                    (blkBldr, value) -> VARCHAR.writeString(blkBldr, readableMetricsToJson(readableMetricsStruct(schema, contentFile, value.type().asStructType()), primitiveFields)));
+            // file_sequence_number


it's rather unclear to me the purpose of the (obvious) comments - i see the old ones as well

to me too, but if this pattern exist then its more unclear not to follow it in the middle

let me remove those in commit before

added 1 commit before with comments removal and extracting some code to method

findinpath · 2026-04-09T08:58:17Z

...n/trino-iceberg/src/main/java/io/trino/plugin/iceberg/system/files/FilesTablePageSource.java

+                writeValueOrNull(pageBuilder, FilesTable.REFERENCED_DATA_FILE_COLUMN_NAME, () -> null, VARCHAR::writeString);
+                writeValueOrNull(pageBuilder, FilesTable.CONTENT_OFFSET_COLUMN_NAME, () -> null, BIGINT::writeLong);
+                writeValueOrNull(pageBuilder, FilesTable.CONTENT_SIZE_IN_BYTES_COLUMN_NAME, () -> null, BIGINT::writeLong);


you could introduce writeNull convenience method to avoid sending () -> null repeatedly

findinpath · 2026-04-09T09:02:27Z

nit:

release notes

Extend $files system table in Iceberg connector with additional columns

Please use the provided pattern in the template to ease up maintainer's work in adding the topic to the release notes PR
#28894

findinpath · 2026-04-09T16:26:56Z

@oskar-szwajkowski do note that there is as well a competing PR from @kaveti #28933

Please reconcile which one should be pushed forward.

Remove redundant comments in FilesTablePageSource Extract code to method

Add file_sequence_number, data_sequence_number, referenced_data_file, pos, manifest_location, first_row_id, content_offset, and content_size_in_bytes columns to the Iceberg $files system table. These columns expose additional metadata from Iceberg manifest entries, including delete-file-specific field like referenced_data_file and content_offset which are populated for deletion vectors.

chenjian2664

Looks good

chenjian2664 · 2026-04-10T13:30:59Z

docs/src/main/sphinx/connector/iceberg.md

+* - `first_row_id`
+  - `BIGINT`
+  - The ID of the first row in the data file.
+* - `content_offset`


chenjian2664 · 2026-04-10T13:32:04Z

...n/trino-iceberg/src/main/java/io/trino/plugin/iceberg/system/files/FilesTablePageSource.java

-            // sort_order_id
            writeValueOrNull(pageBuilder, SORT_ORDER_ID_COLUMN_NAME, contentFile::sortOrderId,
                    (blkBldr, value) -> INTEGER.writeLong(blkBldr, value));
-            // readable_metrics


is it intentional?

chenjian2664 · 2026-04-10T13:34:55Z

...n/trino-iceberg/src/main/java/io/trino/plugin/iceberg/system/files/FilesTablePageSource.java

-            // readable_metrics
            writeValueOrNull(pageBuilder, READABLE_METRICS_COLUMN_NAME, () -> metadataSchema.findField(MetricsUtil.READABLE_METRICS),
                    (blkBldr, value) -> VARCHAR.writeString(blkBldr, readableMetricsToJson(readableMetricsStruct(schema, contentFile, value.type().asStructType()), primitiveFields)));
+            writeValueOrNull(pageBuilder, FilesTable.FILE_SEQUENCE_NUMBER_COLUMN_NAME, contentFile::fileSequenceNumber, BIGINT::writeLong);


static import the FilesTable field

chenjian2664 · 2026-04-10T13:40:38Z

docs/src/main/sphinx/connector/iceberg.md

+* - `content_offset`
+  - `BIGINT`
+  - The offset in the file where the content starts. Only set for deletion
+    vectors, `NULL` for data files.
+* - `content_size_in_bytes`
+  - `BIGINT`
+  - The size of the content in bytes. Only set for deletion vectors, `NULL` for
+    data files.


I think the current name is misleading, as the functionality is clearly valuable for data files as well. I understand the intent behind it, but in that case, I would recommend either incorporating "delete" or "deletion" into the name for clarity - otherwise looks confusion with "file_size_in_bytes"

cla-bot bot added the cla-signed label Apr 8, 2026

github-actions bot added the iceberg Iceberg connector label Apr 8, 2026

oskar-szwajkowski force-pushed the osz/add-more-columns-to-iceberg-files-table branch from 24955ad to d446f21 Compare April 8, 2026 18:24

github-actions bot added the docs label Apr 8, 2026

oskar-szwajkowski force-pushed the osz/add-more-columns-to-iceberg-files-table branch from d446f21 to 3711921 Compare April 8, 2026 20:29

ebyhr requested review from chenjian2664, ebyhr, findinpath and raunaqmorarka April 9, 2026 04:19

findinpath reviewed Apr 9, 2026

View reviewed changes

docs/src/main/sphinx/connector/iceberg.md Show resolved Hide resolved

findinpath reviewed Apr 9, 2026

View reviewed changes

findinpath approved these changes Apr 9, 2026

View reviewed changes

oskar-szwajkowski force-pushed the osz/add-more-columns-to-iceberg-files-table branch from 3711921 to a3fb91b Compare April 9, 2026 09:47

oskar-szwajkowski mentioned this pull request Apr 10, 2026

Add fields for $files table delete file deduplication #28933

Open

oskar-szwajkowski added 2 commits April 10, 2026 12:07

Refactor FilesTablePageSource

8249c3d

Remove redundant comments in FilesTablePageSource Extract code to method

oskar-szwajkowski force-pushed the osz/add-more-columns-to-iceberg-files-table branch from a3fb91b to bd68773 Compare April 10, 2026 10:07

chenjian2664 reviewed Apr 10, 2026

View reviewed changes

ebyhr approved these changes Apr 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add more columns to Iceberg $files system table#29044

Add more columns to Iceberg $files system table#29044
oskar-szwajkowski wants to merge 2 commits intotrinodb:masterfrom
oskar-szwajkowski:osz/add-more-columns-to-iceberg-files-table

oskar-szwajkowski commented Apr 8, 2026 •

edited

Loading

Uh oh!

Uh oh!

findinpath Apr 9, 2026

Uh oh!

oskar-szwajkowski Apr 9, 2026

Uh oh!

oskar-szwajkowski Apr 9, 2026

Uh oh!

findinpath Apr 9, 2026

Uh oh!

oskar-szwajkowski Apr 9, 2026

Uh oh!

findinpath commented Apr 9, 2026

Uh oh!

findinpath commented Apr 9, 2026

Uh oh!

chenjian2664 left a comment

Uh oh!

chenjian2664 Apr 10, 2026

Uh oh!

chenjian2664 Apr 10, 2026

Uh oh!

chenjian2664 Apr 10, 2026

Uh oh!

chenjian2664 Apr 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

Conversation

oskar-szwajkowski commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Additional context and related issues

Release notes

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

findinpath commented Apr 9, 2026

Uh oh!

findinpath commented Apr 9, 2026

Uh oh!

chenjian2664 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

oskar-szwajkowski commented Apr 8, 2026 •

edited

Loading