Skip to content

[Improve][log] Add logs in transform/sink error #9271

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: dev
Choose a base branch
from

Conversation

hailin0
Copy link
Member

@hailin0 hailin0 commented May 6, 2025

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

@Hisoka-X
Copy link
Member

Hisoka-X commented May 7, 2025

Please rebase on dev to trigger ci.

@nielifeng nielifeng requested a review from Copilot May 8, 2025 03:08
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR aims to improve error logging in transform and sink operations by wrapping critical method calls in try-catch blocks and logging more detailed error messages.

  • Introduces try-catch blocks in map/flatMap transforms to capture transformation errors.
  • Adds additional error logging for record processing and sink commit/snapshot operations.
  • Refactors conditional checks in transformation flows to log filtered rows.

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file
File Description
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/common/AbstractMultiCatalogMapTransform.java Enhanced error handling in map transform by wrapping transform.map(row) in try-catch and logging errors.
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/common/AbstractMultiCatalogFlatMapTransform.java Similar error logging improvements for flatMap transform operations.
seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/task/flow/TransformFlowLifeCycle.java Added try-catch around record processing and refined the logic to check for empty output lists.
seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/task/flow/SinkFlowLifeCycle.java Inserted additional error logging when record reception fails.
seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSinkWriter.java Wrapped snapshotState and prepareCommit calls in try-catch blocks with enhanced logging.
seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSinkCommitter.java Added try-catch to log errors during commit operations.
seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSinkAggregatedCommitter.java Enhanced error handling for aggregated commit operations using try-catch and logging.
Comments suppressed due to low confidence (6)

seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/common/AbstractMultiCatalogMapTransform.java:55

  • [nitpick] The error message contains a grammatical issue ('throw' should be 'threw' or rephrased to 'encountered an error while mapping'). Consider revising it for clarity.
String.format("The transform %s map table %s data throw an error", transform.getPluginName(), tableId);

seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/common/AbstractMultiCatalogFlatMapTransform.java:55

  • [nitpick] The error message should be revised to correct the grammatical error ('throw' should be 'threw' or reworded to improve readability).
String.format("The transform %s map table %s data throw an error", transform.getPluginName(), tableId);

seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSinkWriter.java:227

  • [nitpick] The error message in snapshotState has a grammatical mistake; consider changing it to 'threw an error' or 'encountered an error during snapshotState'.
String.format("table %s snapshotState throw an error", sinkWriterEntry.getKey().getTableIdentifier());

seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSinkWriter.java:270

  • [nitpick] Consider updating the error message in prepareCommit to correct the grammar (e.g., 'threw an error during commit') for better clarity.
String.format("table %s prepareCommit throw an error", sinkWriterEntry.getKey().getTableIdentifier());

seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSinkCommitter.java:64

  • [nitpick] The error message in the commit method contains a grammatical error; please consider revising it to 'threw an error' or another clear phrasing.
String.format("table %s commit throw an error", sinkIdentifier);

seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSinkAggregatedCommitter.java:93

  • [nitpick] The error message here should be rephrased to correct the grammar (for example, 'threw an error during commit').
String.format("table %s commit throw an error", sinkIdentifier);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants