Skip to content

[Improve][Transform-V2] Improve sql transform exception to locate error expression #9227

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 28 commits into from
May 28, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
75e8310
[feature] throw sql transform exception,locate error expression
Apr 25, 2025
38eef59
[improve] throw TransformException
Apr 25, 2025
e461da3
[improve] add test case
Apr 28, 2025
59d4fda
[improve] improve code
May 13, 2025
f786dac
Merge branch 'dev' into bugfix-9125
May 14, 2025
be184d1
[improve] remove SqlTransformErrorCode\SqlTrasformException, reuse Tr…
May 15, 2025
d76b087
[improve] 1.add where statement test case
May 16, 2025
683530c
[improve] add the exception into cause stack
May 19, 2025
c433c94
Merge remote-tracking branch 'origin/dev' into bugfix-9125
May 19, 2025
aeeddb7
[improve] modify throw the sqlTransformException
May 20, 2025
bfe9ef9
[improve] add validation of expression values in cause exception
May 20, 2025
8efc95b
Merge branch 'dev' into bugfix-9125
May 22, 2025
42a11a5
[improve] code format test
May 22, 2025
309ade7
[improve] code format test
May 22, 2025
e88bc6e
[feature] throw sql transform exception,locate error expression
Apr 25, 2025
cba7ad7
[improve] throw TransformException
Apr 25, 2025
276eea7
[improve] add test case
Apr 28, 2025
a0649d8
[improve] improve code
May 13, 2025
fd1eee7
[improve] remove SqlTransformErrorCode\SqlTrasformException, reuse Tr…
May 15, 2025
259dcf3
[improve] 1.add where statement test case
May 16, 2025
f1c79a3
[improve] add the exception into cause stack
May 19, 2025
8b59d9f
[improve] modify throw the sqlTransformException
May 20, 2025
41c3ac1
[improve] add validation of expression values in cause exception
May 20, 2025
c70781a
[improve] code format test
May 22, 2025
0065e76
[improve] code format test
May 22, 2025
16c5ee6
Merge remote-tracking branch 'origin/bugfix-9125' into bugfix-9125
May 27, 2025
58bce88
Merge branch 'dev' into bugfix-9125
May 27, 2025
a805e26
[improve] the expression exception is wraped
May 28, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ public void testGetErrorInfo() throws ExecutionException, InterruptedException {

JobResult result = clientJobProxy.getJobResultCache();
Assertions.assertEquals(result.getStatus(), JobStatus.FAILED);
Assertions.assertTrue(result.getError().startsWith("java.lang.NumberFormatException"));
Assertions.assertTrue(result.getError().contains("java.lang.NumberFormatException"));
}
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,13 @@
import java.util.List;
import java.util.Map;

import static org.apache.seatunnel.transform.exception.TransformCommonErrorCode.EXPRESSION_EXECUTE_ERROR;
import static org.apache.seatunnel.transform.exception.TransformCommonErrorCode.INPUT_FIELDS_NOT_FOUND;
import static org.apache.seatunnel.transform.exception.TransformCommonErrorCode.INPUT_FIELD_NOT_FOUND;
import static org.apache.seatunnel.transform.exception.TransformCommonErrorCode.INPUT_TABLE_NOT_FOUND;
import static org.apache.seatunnel.transform.exception.TransformCommonErrorCode.METADATA_FIELDS_NOT_FOUND;
import static org.apache.seatunnel.transform.exception.TransformCommonErrorCode.METADATA_MAPPING_FIELD_EXISTS;
import static org.apache.seatunnel.transform.exception.TransformCommonErrorCode.WHERE_STATEMENT_ERROR;

/** The common error of SeaTunnel transform. Please refer {@link CommonError} */
public class TransformCommonError {
Expand Down Expand Up @@ -67,4 +69,16 @@ public static TransformException cannotFindInputTableError(String transform, Str
params.put("transform", transform);
return new TransformException(INPUT_TABLE_NOT_FOUND, params);
}

public static TransformException sqlExpressionError(String expression, Throwable cause) {
Map<String, String> params = new HashMap<>();
params.put("expression", expression);
return new TransformException(EXPRESSION_EXECUTE_ERROR, params, cause);
}

public static TransformException sqlWhereStatementError(String wherebody, Throwable cause) {
Map<String, String> params = new HashMap<>();
params.put("wherebody", wherebody);
return new TransformException(WHERE_STATEMENT_ERROR, params, cause);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,12 @@ public enum TransformCommonErrorCode implements SeaTunnelErrorCode {
"The metadata mapping field '<field>' of '<transform>' transform already exists in upstream schema"),
INPUT_TABLE_NOT_FOUND(
"TRANSFORM_COMMON-05",
"The input table '<table>' of '<transform>' transform not found in upstream schema");
"The input table '<table>' of '<transform>' transform not found in upstream schema"),
EXPRESSION_EXECUTE_ERROR(
"TRANSFORM_COMMON-06", "The expression '<expression>' of SQL transform execute failed"),
WHERE_STATEMENT_ERROR(
"TRANSFORM_COMMON-07",
"The where statement '<wherebody>' of SQL transform execute failed");

private final String code;
private final String description;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,4 +30,9 @@ public TransformException(SeaTunnelErrorCode seaTunnelErrorCode, String errorMes
TransformException(SeaTunnelErrorCode seaTunnelErrorCode, Map<String, String> params) {
super(seaTunnelErrorCode, params);
}

TransformException(
SeaTunnelErrorCode seaTunnelErrorCode, Map<String, String> params, Throwable cause) {
super(seaTunnelErrorCode, params, cause);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import org.apache.seatunnel.api.table.type.SeaTunnelRow;
import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
import org.apache.seatunnel.common.exception.CommonErrorCodeDeprecated;
import org.apache.seatunnel.transform.exception.TransformCommonError;
import org.apache.seatunnel.transform.exception.TransformException;
import org.apache.seatunnel.transform.sql.SQLEngine;

Expand Down Expand Up @@ -240,9 +241,13 @@ public List<SeaTunnelRow> transformBySQL(SeaTunnelRow inputRow, SeaTunnelRowType
Object[] inputFields = scanTable(inputRow);

// Filter
boolean retain = zetaSQLFilter.executeFilter(selectBody.getWhere(), inputFields);
if (!retain) {
return null;
try {
boolean retain = zetaSQLFilter.executeFilter(selectBody.getWhere(), inputFields);
if (!retain) {
return null;
}
} catch (Exception e) {
throw TransformCommonError.sqlWhereStatementError(selectBody.getWhere().toString(), e);
}

// Project
Expand Down Expand Up @@ -280,8 +285,12 @@ private Object[] project(Object[] inputFields) {
}
} else {
Expression expression = selectItem.getExpression();
fields[idx] = zetaSQLFunction.computeForValue(expression, inputFields);
idx++;
try {
fields[idx] = zetaSQLFunction.computeForValue(expression, inputFields);
idx++;
} catch (Exception e) {
throw TransformCommonError.sqlExpressionError(expression.toString(), e);
}
}
}
return fields;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -397,8 +397,11 @@ public void tesCastBooleanClausesWithField() {
new SeaTunnelRow(new Object[] {Integer.valueOf(1), 3, "false"}));
} catch (Exception e) {
Assertions.assertEquals(
"ErrorCode:[COMMON-05], ErrorDescription:[Unsupported operation] - Unsupported CAST AS Boolean: 3",
"ErrorCode:[TRANSFORM_COMMON-06], ErrorDescription:[The expression 'cast(`int` AS boolean)' of SQL transform execute failed]",
Copy link
Preview

Copilot AI May 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The expected error message uses lowercase cast, but the actual exception uses CAST (uppercase) from the expression's toString(). Update this assertion to match CAST for consistency.

Suggested change
"ErrorCode:[TRANSFORM_COMMON-06], ErrorDescription:[The expression 'cast(`int` AS boolean)' of SQL transform execute failed]",
"ErrorCode:[TRANSFORM_COMMON-06], ErrorDescription:[The expression 'CAST(`int` AS boolean)' of SQL transform execute failed]",

Copilot uses AI. Check for mistakes.

e.getMessage());
Assertions.assertEquals(
"ErrorCode:[COMMON-05], ErrorDescription:[Unsupported operation] - Unsupported CAST AS Boolean: 3",
e.getCause().getMessage());
throw e;
}
});
Expand All @@ -411,8 +414,11 @@ public void tesCastBooleanClausesWithField() {
new SeaTunnelRow(new Object[] {Integer.valueOf(1), 0, "false333"}));
} catch (Exception e) {
Assertions.assertEquals(
"ErrorCode:[COMMON-05], ErrorDescription:[Unsupported operation] - Unsupported CAST AS Boolean: false333",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new exception looks like lose expression value in message. We should fix it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

already complete

"ErrorCode:[TRANSFORM_COMMON-06], ErrorDescription:[The expression 'cast(`string` AS boolean)' of SQL transform execute failed]",
Copy link
Preview

Copilot AI May 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The expected error message uses lowercase cast, but the actual exception uses CAST. Update this assertion to expect CAST(string AS boolean).

Suggested change
"ErrorCode:[TRANSFORM_COMMON-06], ErrorDescription:[The expression 'cast(`string` AS boolean)' of SQL transform execute failed]",
"ErrorCode:[TRANSFORM_COMMON-06], ErrorDescription:[The expression 'CAST(`string` AS boolean)' of SQL transform execute failed]",

Copilot uses AI. Check for mistakes.

e.getMessage());
Assertions.assertEquals(
"ErrorCode:[COMMON-05], ErrorDescription:[Unsupported operation] - Unsupported CAST AS Boolean: false333",
e.getCause().getMessage());
throw e;
}
});
Expand Down Expand Up @@ -441,4 +447,55 @@ public void tesBooleanField() {
Assertions.assertEquals(true, result.get(0).getField(1));
Assertions.assertEquals(false, result.get(0).getField(2));
}

@Test
public void testExpressionErrorField() {
String tableName = "test";
String[] fields = new String[] {"FIELD1", "FIELD2", "FIELD3"};
SeaTunnelDataType[] fieldTypes =
new SeaTunnelDataType[] {
BasicType.INT_TYPE, BasicType.DOUBLE_TYPE, BasicType.STRING_TYPE
};
CatalogTable table =
CatalogTableUtil.getCatalogTable(
tableName, new SeaTunnelRowType(fields, fieldTypes));
String sqlQuery =
"select "
+ "CAST(`FIELD1` AS STRING) AS FIELD1, "
+ "CAST(`FIELD1` AS decimal(22,4)) AS FIELD2, "
+ "CAST(`FIELD3` AS decimal(22,0)) AS FIELD3 "
+ "from dual";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should handle where statement too.


ReadonlyConfig config = ReadonlyConfig.fromMap(Collections.singletonMap("query", sqlQuery));
SQLTransform sqlTransform = new SQLTransform(config, table);
Assertions.assertThrows(
TransformException.class,
() -> {
try {
sqlTransform.transformRow(
new SeaTunnelRow(new Object[] {1, 123.123, "true"}));
} catch (Exception e) {
Assertions.assertEquals(
"ErrorCode:[TRANSFORM_COMMON-06], ErrorDescription:[The expression 'CAST(`FIELD3` AS decimal (22, 0))' of SQL transform execute failed]",
Copy link
Preview

Copilot AI May 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spacing in decimal (22, 0) may not match the expression's toString() output (likely decimal(22,0)). Adjust the expected string to CAST(FIELD3 AS decimal(22,0)) to align with the actual error message.

Suggested change
"ErrorCode:[TRANSFORM_COMMON-06], ErrorDescription:[The expression 'CAST(`FIELD3` AS decimal (22, 0))' of SQL transform execute failed]",
"ErrorCode:[TRANSFORM_COMMON-06], ErrorDescription:[The expression 'CAST(`FIELD3` AS decimal(22,0))' of SQL transform execute failed]",

Copilot uses AI. Check for mistakes.

e.getMessage());
throw e;
}
});
sqlQuery = "select * from dual where FIELD1/0 > 10";
config = ReadonlyConfig.fromMap(Collections.singletonMap("query", sqlQuery));
SQLTransform sqlTransform2 = new SQLTransform(config, table);
Assertions.assertThrows(
TransformException.class,
() -> {
try {
sqlTransform2.transformRow(
new SeaTunnelRow(new Object[] {1, 123.123, "true"}));
} catch (Exception e) {
Assertions.assertEquals(
"ErrorCode:[TRANSFORM_COMMON-07], ErrorDescription:[The where statement 'FIELD1 / 0 > 10' of SQL transform execute failed]",
e.getMessage());
throw e;
}
});
}
}