Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feat][Connector-v2][Iceberg]support filter conditions in iceberg source #9095

Open
wants to merge 14 commits into
base: dev
Choose a base branch
from

Conversation

litiliu
Copy link
Contributor

@litiliu litiliu commented Apr 1, 2025

Purpose of this pull request

close #9081

Does this PR introduce any user-facing change?

Updated Doc

How was this patch tested?

Unit test and E2E case added

Check list

@litiliu litiliu changed the title [Feat][Connector-v2][Iceberg]support filter conditions in icberg source [Feat][Connector-v2][Iceberg]support filter conditions in iceberg source Apr 1, 2025
@hailin0 hailin0 requested a review from Copilot April 1, 2025 09:08
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This pull request enhances the Iceberg connector by adding support for filtering data using SQL-like WHERE clauses. Key changes include new tests for SQL-to-Iceberg expression conversion, updates to the expression converter to handle various SQL expressions, and documentation updates for the new "where_clause" option.

Reviewed Changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.

File Description
connector-iceberg/src/test/java/org/apache/seatunnel/connectors/seatunnel/iceberg/config/SQLToIcebergExprConverterTest.java Added comprehensive tests covering various SQL expressions.
connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/config/SQLToIcebergExprConverter.java Implements the conversion from SQL to Iceberg expressions with potential issue in NullValue handling.
connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/config/IcebergSourceOptions.java Introduces the new "where_clause" option.
docs/en/connector-v2/source/Iceberg.md Updates documentation to include the new filtering option.
Files not reviewed (1)
  • seatunnel-connectors-v2/connector-iceberg/pom.xml: Language not supported

@hailin0
Copy link
Member

hailin0 commented Apr 1, 2025

Please add a testcase

@github-actions github-actions bot added the e2e label Apr 2, 2025
@litiliu litiliu marked this pull request as draft April 2, 2025 09:52
@litiliu litiliu marked this pull request as ready for review April 3, 2025 05:43
@litiliu litiliu requested a review from hailin0 April 3, 2025 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature][Connector-v2][Iceberg][Source]Support filter expression in the source config
2 participants