Add optional snapshot summary statistics for AWS Redshift compatibility #634
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This patch adds the required snapshot summary statistics (total-records, total-data-files, total-files-size, etc.) that are needed for Amazon Redshift and other query engines to read DuckDB-created Iceberg tables. Previously, DuckDB only set the "operation" field in the snapshot summary, which caused Redshift to fail with: "Error parsing table metadata. Required field total-records missing." Changes: - Add IcebergSnapshotSummary struct to track cumulative and delta statistics - Update ToRESTObject() to serialize statistics to additional_properties - Update AddSnapshot() and AddUpdateSnapshot() to compute statistics from manifest entries and previous snapshot totals - Update ParseSnapshot() to parse statistics from existing metadata files Fixes: duckdb#633 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Per Iceberg spec Appendix F, snapshot summary fields like total-records are optional. However, AWS Redshift requires these fields and fails with "Required field total-records missing" when they're absent. This change serializes the additional_properties from IcebergSnapshotSummary to the metadata JSON, which includes statistics like total-records, total-data-files, added-records, etc. Reference: https://iceberg.apache.org/spec/#appendix-f-optional-snapshot-summary-fields 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
total-records,total-data-files,added-records) to metadata JSONFixes #633
Problem
AWS Redshift fails to read DuckDB-created Iceberg tables with error:
Root Cause
DuckDB's
IcebergSnapshotSummarystruct already computes these statistics inadditional_properties, but they weren't being serialized to the JSON output. TheCommitTableToJSONfunction only wrote theoperationfield.Solution
Added a loop to serialize all
additional_propertiesfrom the snapshot summary:Iceberg Spec Reference
Per Iceberg spec Appendix F, these fields are optional. However, AWS Redshift requires them for compatibility.
Test plan
🤖 Generated with Claude Code