Skip to content

[Feature][Transform] Add table filter transform #9189

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: dev
Choose a base branch
from

Conversation

hailin0
Copy link
Member

@hailin0 hailin0 commented Apr 16, 2025

Purpose of this pull request

[Transform] Add table filter transform

Does this PR introduce any user-facing change?

No

How was this patch tested?

Add

Check list

@hailin0 hailin0 requested a review from Copilot April 16, 2025 15:04
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This pull request adds a new TableFilter transform plugin that supports filtering tables based on database, schema, and table name patterns. Additionally, the PR removes legacy shuffle-related code from the engine and updates documentation and end-to-end tests to cover the new transform.

Reviewed Changes

Copilot reviewed 18 out of 20 changed files in this pull request and generated no comments.

Show a summary per file
File Description
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/table/TableFilterTransformFactory.java Implements the factory for the new TableFilter transform.
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/table/TableFilterTransform.java Defines the transform that conditionally forwards rows based on the include flag.
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/table/TableFilterMultiCatalogTransform.java Supports multi-catalog transformation using the TableFilter configuration.
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/table/TableFilterConfig.java Provides configuration options for the TableFilter transform.
docs/*/transform-v2/table-filter.md Updates documentation to describe the new TableFilter plugin.
seatunnel-e2e/seatunnel-transforms-v2-e2e-part-2/src/test/java/org/apache/seatunnel/e2e/transform/TestTableFilterIT.java Adds e2e test coverage for the TableFilter transform.
Several engine files Remove legacy shuffle-related classes and shuffle edge generation methods.
Files not reviewed (2)
  • plugin-mapping.properties: Language not supported
  • seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-2/src/test/resources/table_filter_multi_table.conf: Language not supported
Comments suppressed due to low confidence (3)

seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/table/TableFilterConfig.java:125

  • Since the default value for pattern_mode ensures it is never null, this precondition always passes and may not effectively enforce that a filtering pattern is provided. Consider revising the check to require at least one of database_pattern, schema_pattern, or table_pattern to be non-null.
Preconditions.checkArgument(filterConfig.getDatabasePattern() != null || filterConfig.getSchemaPattern() != null || filterConfig.getTablePattern() != null || filterConfig.getPatternMode() != null, "At least one of database_pattern, schema_pattern, table_pattern or pattern_mode must be specified.");

docs/zh/transform-v2/table-filter.md:1

  • The title 'TableMerge' appears inconsistent with the plugin functionality and name ('TableFilter'); please update it to 'TableFilter' to avoid confusion.
# TableMerge

docs/en/transform-v2/table-filter.md:1

  • The documentation title 'TableMerge' is misleading given the plugin is for filtering tables; consider renaming it to 'TableFilter' for clarity.
# TableMerge

@@ -80,27 +72,21 @@ public ExecutionPlan generate() {
Set<ExecutionEdge> executionEdges = generateExecutionEdges(logicalPlan.getEdges());
log.debug("Phase 1: generate execution edge list {}", executionEdges);

executionEdges = generateShuffleEdges(executionEdges);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delete unused code for shuffle

@nielifeng nielifeng requested a review from Copilot April 17, 2025 01:16
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 18 out of 20 changed files in this pull request and generated no comments.

Files not reviewed (2)
  • plugin-mapping.properties: Language not supported
  • seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-2/src/test/resources/table_filter_multi_table.conf: Language not supported
Comments suppressed due to low confidence (3)

docs/zh/transform-v2/table-filter.md:1

  • [nitpick] The title 'TableMerge' seems inconsistent with the transform's functionality, which is table filtering. Please consider renaming it to 'TableFilter'.
# TableMerge

docs/en/transform-v2/table-filter.md:1

  • [nitpick] The title 'TableMerge' appears to be a copy-paste error. It should be updated to 'TableFilter' to accurately reflect the transform's purpose.
# TableMerge

seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/table/TableFilterConfig.java:124

  • The precondition check includes 'pattern_mode' in its condition even though it always has a default value (INCLUDE). This makes the check always pass, which may be misleading. Consider revising the precondition to require at least one filtering pattern to be specified.
Preconditions.checkArgument(...)

@github-actions github-actions bot added the core SeaTunnel core module label Apr 17, 2025
@hailin0 hailin0 marked this pull request as ready for review April 17, 2025 10:52
@hailin0 hailin0 mentioned this pull request Apr 21, 2025
3 tasks
@hailin0
Copy link
Member Author

hailin0 commented Apr 23, 2025

PTAL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants