Skip to content

feat(iceberg_fdw): [WRA-16] add schema evolution support#590

Merged
burmecia merged 3 commits intomainfrom
bo/feat/iceberg-schema-evolution
Mar 24, 2026
Merged

feat(iceberg_fdw): [WRA-16] add schema evolution support#590
burmecia merged 3 commits intomainfrom
bo/feat/iceberg-schema-evolution

Conversation

@burmecia
Copy link
Copy Markdown
Member

What kind of change does this PR introduce?

This PR is to add schema evolution support to the Iceberg FDW and upgrades iceberg-rs from 0.6.0 to 0.8.0.

What is the current behavior?

  • Reading an Iceberg table that has undergone schema evolution (e.g. columns added after initial writes) fails with DataInvalid: missing_fields. The 0.6.0 parquet reader does not tolerate data files that predate a newly-added optional column.
  • Schema evolution is listed as unsupported in the docs.

What is the new behavior?

  • Schema-evolved tables are read correctly. Rows written before a column was added return NULL for that column; rows written after return the actual value — consistent with the Iceberg spec.
  • Filter pushdown on evolved columns works as expected.
  • The dependency upgrade to iceberg-rs 0.8.0 to support this new feature

Additional context

N/A

Copilot AI review requested due to automatic review settings March 24, 2026 01:52
@burmecia burmecia changed the title feat(iceberg_fdw): add schema evolution support feat(iceberg_fdw): [WRA-16] add schema evolution support Mar 24, 2026
@burmecia burmecia added the enhancement New feature or request label Mar 24, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds schema evolution support to iceberg_fdw by upgrading to iceberg-rs 0.8.0 and wiring schema resolution/pushdown/writer changes needed to correctly read evolved Iceberg tables (older rows returning NULL for newly-added optional columns), with accompanying docs and tests.

Changes:

  • Upgrade iceberg / catalog crates (and Arrow/Parquet) to newer versions required for schema evolution tolerant reads.
  • Add schema selection by schema_id (optional) and adjust predicate pushdown to operate on the resolved schema.
  • Extend test/seed data to validate evolved-schema reads and pushdown on newly-added columns.

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
wrappers/src/fdw/iceberg_fdw/writer.rs Adapts to upstream location generator API change (partition key-aware signature).
wrappers/src/fdw/iceberg_fdw/pushdown.rs Changes pushdown API to accept an explicit Schema, enabling evolution-aware predicate building.
wrappers/src/fdw/iceberg_fdw/mod.rs Adds SchemaNotFound error variant for explicit schema_id resolution failures.
wrappers/src/fdw/iceberg_fdw/iceberg_fdw.rs Implements schema_id resolution on scan, updates Arrow schema conversion, upgrades catalog construction, and adjusts writer pipeline for new iceberg-rs APIs.
wrappers/src/fdw/iceberg_fdw/tests.rs Adds schema evolution integration assertions and validates default-schema behavior when schema_id is dropped.
wrappers/src/fdw/iceberg_fdw/README.md Bumps FDW version and changelog entry.
wrappers/dockerfiles/s3/iceberg_seed.py Seeds a schema-evolved Iceberg table used by integration tests.
wrappers/Cargo.toml Upgrades iceberg/arrow/parquet dependencies to required versions.
docs/catalog/iceberg.md Documents schema evolution support and removes the prior “unsupported” limitation.
Cargo.lock Lockfile updates reflecting the dependency upgrades.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

burmecia and others added 2 commits March 24, 2026 14:19
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@burmecia burmecia merged commit 3edd0b1 into main Mar 24, 2026
7 checks passed
@burmecia burmecia deleted the bo/feat/iceberg-schema-evolution branch March 24, 2026 03:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request iceberg

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants