[Fix][Transform-v2] Fix custom udf throw exception #9195

hawk9821 · 2025-04-17T08:41:29Z

Purpose of this pull request

custom udf throw Exception. remove seatunnel-transform-v2.jar from connecots to lib solve the problem .

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/seatunnel/transform/sql/zeta/ZetaUDF
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
	at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
	at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:97)
	at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370)
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	at java.lang.Iterable.forEach(Iterable.java:74)
	at org.apache.seatunnel.transform.sql.zeta.ZetaSQLEngine.init(ZetaSQLEngine.java:88)
	at org.apache.seatunnel.transform.sql.SQLTransform.open(SQLTransform.java:94)
	at org.apache.seatunnel.transform.sql.SQLTransform.tryOpen(SQLTransform.java:103)
	at org.apache.seatunnel.transform.sql.SQLTransform.transformTableSchema(SQLTransform.java:115)
	at org.apache.seatunnel.transform.common.AbstractSeaTunnelTransform.transformCatalogTable(AbstractSeaTunnelTransform.java:69)
	at org.apache.seatunnel.transform.common.AbstractSeaTunnelTransform.getProducedCatalogTable(AbstractSeaTunnelTransform.java:54)
	at org.apache.seatunnel.transform.common.AbstractMultiCatalogTransform.lambda$new$3(AbstractMultiCatalogTransform.java:86)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
	at org.apache.seatunnel.transform.common.AbstractMultiCatalogTransform.<init>(AbstractMultiCatalogTransform.java:88)
	at org.apache.seatunnel.transform.common.AbstractMultiCatalogFlatMapTransform.<init>(AbstractMultiCatalogFlatMapTransform.java:33)
	at org.apache.seatunnel.transform.sql.SQLMultiCatalogFlatMapTransform.<init>(SQLMultiCatalogFlatMapTransform.java:32)
	at org.apache.seatunnel.transform.sql.SQLTransformFactory.lambda$createTransform$0(SQLTransformFactory.java:51)
	at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareMultiTableTransform(FactoryUtil.java:445)
	at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseTransform(MultipleTableJobConfigParser.java:459)
	at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseTransforms(MultipleTableJobConfigParser.java:408)
	at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:232)
	at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:123)
	at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:191)
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:165)
	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
	at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.seatunnel.transform.sql.zeta.ZetaUDF
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 49 more

Does this PR introduce any user-facing change?

How was this patch tested?

e2e case : ExampleUdfIT

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config
Update the release-note.

hailin0 · 2025-04-17T10:56:50Z

seatunnel-dist/src/main/assembly/assembly-bin.xml

+                <include>org.apache.seatunnel:seatunnel-transforms-v2:jar</include>
+            </includes>
+            <outputFileNameMapping>${artifact.file.name}</outputFileNameMapping>
+            <outputDirectory>/lib</outputDirectory>


Why? This conflicts with pr #7218

pr #7218 does not consider the scenarios of custom udf , moved seatunnel-transform-v2.jar to connectors will throw exception.

The core reason is NoClassDefFoundError org.apache.seatunnel.transform.sql.zeta.ZetaUDF. It would be more appropriate to move interface org.apache.seatunnel.transform.sql.zeta.ZetaUDF to the api module

cc @liunaijie @Hisoka-X

Copilot

Pull Request Overview

This pull request addresses a NoClassDefFoundError in custom UDFs by updating the import paths for ZetaUDF and relocating the transform UDF jar from the connectors directory to the lib directory.

Update import statements in several UDF-related classes to use the new org.apache.seatunnel.api.transform.sql.ZetaUDF package
Adjust the container copy logic to load the transform UDF jar from the lib directory
Update the API package for ZetaUDF accordingly

Reviewed Changes

Copilot reviewed 9 out of 13 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/functions/udf/DesEncrypt.java	Updated ZetaUDF import to the new API package
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/functions/udf/DesDecrypt.java	Updated ZetaUDF import to the new API package
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/ZetaSQLType.java	Added import for ZetaUDF from the API package
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/ZetaSQLFunction.java	Added import for ZetaUDF from the API package
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/ZetaSQLEngine.java	Added import for ZetaUDF from the API package
seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-udf/src/main/java/org/apache/seatunnel/e2e/transform/udf/ExampleUdf.java	Provides an example implementation for custom UDF testing
seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-udf/src/test/java/org/apache/seatunnel/e2e/transform/udf/ExampleUdfIT.java	End-to-end integration test for the custom UDF
seatunnel-e2e/seatunnel-e2e-common/src/test/java/org/apache/seatunnel/e2e/common/util/ContainerUtil.java	Updated the container copy logic to move the UDF jar to the lib directory
seatunnel-api/src/main/java/org/apache/seatunnel/api/transform/sql/ZetaUDF.java	Changed the package to the new API namespace for ZetaUDF

Files not reviewed (4)

seatunnel-e2e/seatunnel-transforms-v2-e2e/pom.xml: Language not supported
seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-udf/pom.xml: Language not supported
seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-udf/src/test/resources/custom_udf.conf: Language not supported
seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-udf/pom.xml: Language not supported

Hisoka-X · 2025-04-21T04:10:07Z

seatunnel-api/src/main/java/org/apache/seatunnel/api/transform/sql/ZetaUDF.java

@@ -15,7 +15,7 @@
 * limitations under the License.
 */

-package org.apache.seatunnel.transform.sql.zeta;
+package org.apache.seatunnel.api.transform.sql;


If we move UDF to api module. Why we still need move seatunnel-transform-v2.jar to libs?

not needed. assembly-bin.xml is rollbacked

liunaijie · 2025-04-22T16:10:52Z

Thanks @hawk9821 point this issue, this is a bug, good catch.

It would be more appropriate to move interface org.apache.seatunnel.transform.sql.zeta.ZetaUDF to the api module

I tested with this fix, it works. But I don’t understand why these changes can be work. Can you give me some hint? Thanks.

BTW, when do this update, it can impact the user that has implment udf. it need update the udf lib, update the class full name, we need describe this in the document or release note (Breaking Change).

hawk9821 · 2025-04-24T00:52:54Z

Thanks @hawk9821 point this issue, this is a bug, good catch.

It would be more appropriate to move interface org.apache.seatunnel.transform.sql.zeta.ZetaUDF to the api module

I tested with this fix, it works. But I don’t understand why these changes can be work. Can you give me some hint? Thanks.

BTW, when do this update, it can impact the user that has implment udf. it need update the udf lib, update the class full name, we need describe this in the document or release note (Breaking Change).

The compatibility issue was indeed not considered. It has been fixed in the way of moving transform-v2.jar to the lib .

Hisoka-X · 2025-04-24T04:12:12Z

seatunnel-dist/src/main/assembly/assembly-bin.xml

+            <outputDirectory>/lib</outputDirectory>
+            <scope>provided</scope>
+        </dependencySet>
+
        <!-- ============ Connectors Jars And Transforms V2 Jar ============  -->


Suggested change

Hisoka-X · 2025-04-24T04:13:10Z

...ansforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/functions/udf/ZetaUDF.java

@@ -15,7 +15,7 @@
 * limitations under the License.
 */

-package org.apache.seatunnel.transform.sql.zeta;
+package org.apache.seatunnel.transform.sql.zeta.functions.udf;


why move the package of ZetaUDF?

hailin0 · 2025-04-24T04:13:57Z

Need to delete transform from plugin-mapping.properties?
https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties #L148

github-actions bot added core SeaTunnel core module e2e labels Apr 17, 2025

hawk9821 changed the title ~~Custom udf~~ [BUG][Transform-v2]Custom udf throw exception Apr 17, 2025

hailin0 reviewed Apr 17, 2025

View reviewed changes

hawk9821 force-pushed the custom_udf branch from ee93818 to 0ef3480 Compare April 17, 2025 13:36

github-actions bot added Transform-v2 api and removed core SeaTunnel core module labels Apr 17, 2025

nielifeng requested a review from Copilot April 18, 2025 01:31

Copilot AI reviewed Apr 18, 2025

View reviewed changes

Hisoka-X changed the title ~~[BUG][Transform-v2]Custom udf throw exception~~ [Fix][Transform-v2] Fix custom udf throw exception Apr 21, 2025

Hisoka-X reviewed Apr 21, 2025

View reviewed changes

github-actions bot added document CI&CD core SeaTunnel core module Zeta connectors-v2 Zeta Rest API labels Apr 22, 2025

hawk9821 force-pushed the custom_udf branch from a110274 to eebf18c Compare April 22, 2025 07:14

hailin0 mentioned this pull request Apr 22, 2025

SeaTunnel Community Meeting #9174

Closed

3 tasks

hawk9821 added 3 commits April 23, 2025 12:27

[Bug][Transform]fix custom udf throw exception

4bb899b

[Bug][Transform]fix custom udf throw exception

e95e2c6

[Bug][Transform]fix custom udf throw exception

c81ae15

hawk9821 force-pushed the custom_udf branch from eebf18c to c81ae15 Compare April 23, 2025 04:27

github-actions bot removed document CI&CD Zeta connectors-v2 Zeta Rest API labels Apr 23, 2025

github-actions bot removed the api label Apr 23, 2025

liunaijie approved these changes Apr 24, 2025

View reviewed changes

github-actions bot added approved reviewed labels Apr 24, 2025

Hisoka-X approved these changes Apr 24, 2025

View reviewed changes

Hisoka-X requested changes Apr 24, 2025

View reviewed changes

github-actions bot removed approved reviewed labels Apr 24, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fix][Transform-v2] Fix custom udf throw exception #9195

[Fix][Transform-v2] Fix custom udf throw exception #9195

hawk9821 commented Apr 17, 2025 •

edited

Loading

hailin0 Apr 17, 2025

hawk9821 Apr 17, 2025

hawk9821 Apr 17, 2025

hailin0 Apr 17, 2025

Copilot AI left a comment

Hisoka-X Apr 21, 2025

hawk9821 Apr 21, 2025

liunaijie commented Apr 22, 2025

hawk9821 commented Apr 24, 2025

Hisoka-X Apr 24, 2025

Hisoka-X Apr 24, 2025

hailin0 commented Apr 24, 2025

	<!-- ============ Connectors Jars And Transforms V2 Jar ============ -->
	<!-- ============ Connectors Jars ============ -->

[Fix][Transform-v2] Fix custom udf throw exception #9195

Are you sure you want to change the base?

[Fix][Transform-v2] Fix custom udf throw exception #9195

Conversation

hawk9821 commented Apr 17, 2025 • edited Loading

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liunaijie commented Apr 22, 2025

hawk9821 commented Apr 24, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hailin0 commented Apr 24, 2025

hawk9821 commented Apr 17, 2025 •

edited

Loading