-
Notifications
You must be signed in to change notification settings - Fork 3.4k
Populate snapshot summary in iceberg ConnectorOutputMetadata #27961
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Populate snapshot summary in iceberg ConnectorOutputMetadata #27961
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR enhances Iceberg table operations by populating snapshot summary metadata in ConnectorOutputMetadata for CTAS, INSERT, and REFRESH MATERIALIZED VIEW queries. The metadata includes useful metrics like added records, data files, file sizes, and partition counts that are now visible in the TableFinish operator summary.
Changes:
- Created new
IcebergCommitMetadataclass to wrap Iceberg snapshot summary information - Modified
finishInsertto return snapshot summary instead of file paths - Modified
finishRefreshMaterializedViewto return snapshot summary instead of file paths - Added test coverage for INSERT, CTAS, and REFRESH MV operations
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| IcebergCommitMetadata.java | New class implementing ConnectorOutputMetadata to hold snapshot summary metrics |
| IcebergMetadata.java | Modified finishInsert and finishRefreshMaterializedView to capture and return snapshot summaries; removed HiveWrittenPartitions dependency |
| BaseIcebergConnectorTest.java | Added testCommitMetrics to verify snapshot metadata is properly populated for INSERT, CTAS, and REFRESH MV operations |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorTest.java
Outdated
Show resolved
Hide resolved
plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergCommitMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java
Outdated
Show resolved
Hide resolved
47e2139 to
33531ab
Compare
plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java
Outdated
Show resolved
Hide resolved
plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java
Outdated
Show resolved
Hide resolved
33531ab to
5001f13
Compare
5001f13 to
f63c023
Compare
Description
Adds useful information to TableFinish operator summary of CTAS, INSERT and REFRESH MV queries
e.g.
Additional context and related issues
Release notes
(x) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text: