[improve] Adjust CosFile connector instantiation by misi1987107 · Pull Request #9121 · apache/seatunnel

misi1987107 · 2025-04-08T06:55:47Z

Purpose of this pull request

subtask of #8576

Adjust CosFile connector instantiation

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config
Update the release-note.

Hisoka-X · 2025-04-09T09:08:00Z

...ile-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/BaseFileSink.java

-    @Override
-    public void setTypeInfo(SeaTunnelRowType seaTunnelRowType) {
-        this.seaTunnelRowType = seaTunnelRowType;
-        this.fileSinkConfig = new FileSinkConfig(pluginConfig, seaTunnelRowType);
-    }


Why delete this? We still have some sink should invoke it like OssFileSink, HdfsFileSink etc.

Sorry,I neet to make adjustments according to purpose2 ,move the connector creation logic from Connector to ConnectorFactory.I don't know how to adjust it
(#8576 (comment))

Hisoka-X · 2025-04-09T09:09:24Z

...base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/BaseFileSource.java

+    @Override
+    public List<CatalogTable> getProducedCatalogTables() {
+        return SeaTunnelSource.super.getProducedCatalogTables();
+    }


We should return catalog table in here to replace getProducedType method.

Hisoka-X · 2025-04-09T09:11:38Z

...ain/java/org/apache/seatunnel/connectors/seatunnel/file/cos/source/CosFileSourceFactory.java

+                (SeaTunnelSource<T, SplitT, StateT>)
+                        new CosFileSource(
+                                context.getOptions(),
+                                CatalogTableUtil.buildWithConfig(context.getOptions()));


-1. We should build catalogtable with file sometimes. Please refer https://github.com/apache/seatunnel/pull/9121/files#diff-1f7ebddd2239c9ee909ee66c8f67c119b92f05324519ea5ce40d5aeab98da984L124

This reverts commit a9fc8ca.

misi1987107 · 2025-04-10T01:28:26Z

#8576 (comment)
About purpose2，can i adjust it like this?Please give me some advice. @Hisoka-X @liunaijie

misi1987107 · 2025-04-14T06:17:29Z

I used the factory class to create CosFileSource and CosFileLink connectors, but there is a check method testAllConnectorImplementeFactoryWithUpToDateMethod that does not allow deprecated methods, so I skipped the check.
Please refer
#9121 (comment)
#9121 (comment)

Hisoka-X · 2025-04-15T02:30:20Z

I used the factory class to create CosFileSource and CosFileLink connectors, but there is a check method testAllConnectorImplementeFactoryWithUpToDateMethod that does not allow deprecated methods, so I skipped the check.

Please do not do that. The sub reason of #8576 is make sure all method up to date without deprecated method.

misi1987107 · 2025-04-16T06:09:56Z

I used the factory class to create CosFileSource and CosFileLink connectors, but there is a check method testAllConnectorImplementeFactoryWithUpToDateMethod that does not allow deprecated methods, so I skipped the check.

Please do not do that. The sub reason of #8576 is make sure all method up to date without deprecated method.

If I don't skip the check method testAllConnectorImplementFactoryWithUpToDateMethod(),I must delete the method BaseFileSink.setTypeInfo(),otherwise, it cannot pass through.

Copilot

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Copilot · 2025-04-16T06:56:40Z

...l-dist/src/test/java/org/apache/seatunnel/api/connector/ConnectorSpecificationCheckTest.java

+        // adjusted the connector implementation,not deal with deprecated method yet
+        blockList.add("CosFileSourceFactory");


[nitpick] Consider adding a space after the comma for clarity, e.g., 'connector implementation, not dealing with deprecated method yet'.

Suggested change

// adjusted the connector implementation,not deal with deprecated method yet

blockList.add("CosFileSourceFactory");

// adjusted the connector implementation, not deal with deprecated method yet

Copilot · 2025-04-16T06:56:41Z

...ain/java/org/apache/seatunnel/connectors/seatunnel/file/cos/source/CosFileSourceFactory.java

+    @Override
+    public <T, SplitT extends SourceSplit, StateT extends Serializable>
+            TableSource<T, SplitT, StateT> createSource(TableSourceFactoryContext context) {
+        return () -> (SeaTunnelSource<T, SplitT, StateT>) new CosFileSource(context.getOptions());


Consider adding an annotation to suppress the unchecked cast warning or handling the cast more explicitly to ensure type safety.

Suggested change

return () -> (SeaTunnelSource<T, SplitT, StateT>) new CosFileSource(context.getOptions());

SeaTunnelSource<?, ?, ?> source = new CosFileSource(context.getOptions());

if (!(source instanceof SeaTunnelSource<T, SplitT, StateT>)) {

throw new ClassCastException("The created source is not compatible with the expected type.");

}

return () -> (SeaTunnelSource<T, SplitT, StateT>) source;

[improve] Adjust the instantiation method of CosFile connector

8e86af1

github-actions bot added connectors-v2 file labels Apr 8, 2025

misi added 3 commits April 8, 2025 15:01

[improve] code format

776e8b7

[improve] implements getProducedCatalogTables() method

b99661f

[improve] modify deprecated method

a9fc8ca

Hisoka-X reviewed Apr 9, 2025

View reviewed changes

misi added 3 commits April 9, 2025 19:54

Revert "[improve] modify deprecated method"

c0baac8

This reverts commit a9fc8ca.

[bugfix] remove CosFileSource prepare()

98fac43

[improve] remove CosFileSink prepare()

7b8edfc

[improve] skip deprecated method check

37f68aa

github-actions bot added the core SeaTunnel core module label Apr 14, 2025

nielifeng requested a review from Copilot April 16, 2025 06:55

Copilot AI reviewed Apr 16, 2025

View reviewed changes

misi1987107 closed this May 29, 2025

misi1987107 deleted the feature-file-cos-connector-creation-modify branch June 6, 2025 06:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

[improve] Adjust CosFile connector instantiation#9121

[improve] Adjust CosFile connector instantiation#9121
misi1987107 wants to merge 8 commits intoapache:devfrom
misi1987107:feature-file-cos-connector-creation-modify

misi1987107 commented Apr 8, 2025

Uh oh!

Hisoka-X Apr 9, 2025

Uh oh!

misi1987107 Apr 9, 2025

Uh oh!

Hisoka-X Apr 9, 2025

Uh oh!

Hisoka-X Apr 9, 2025

Uh oh!

misi1987107 commented Apr 10, 2025 •

edited

Loading

Uh oh!

misi1987107 commented Apr 14, 2025

Uh oh!

Hisoka-X commented Apr 15, 2025

Uh oh!

misi1987107 commented Apr 16, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 16, 2025

Uh oh!

Copilot AI Apr 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		// adjusted the connector implementation,not deal with deprecated method yet
		blockList.add("CosFileSourceFactory");

-        return () -> (SeaTunnelSource<T, SplitT, StateT>) new CosFileSource(context.getOptions());
+        SeaTunnelSource<?, ?, ?> source = new CosFileSource(context.getOptions());
+        if (!(source instanceof SeaTunnelSource<T, SplitT, StateT>)) {
+            throw new ClassCastException("The created source is not compatible with the expected type.");
+        }
+        return () -> (SeaTunnelSource<T, SplitT, StateT>) source;

Comments

Conversation

misi1987107 commented Apr 8, 2025

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Uh oh!

Hisoka-X Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

misi1987107 Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

Hisoka-X Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

Hisoka-X Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

misi1987107 commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

misi1987107 commented Apr 14, 2025

Uh oh!

Hisoka-X commented Apr 15, 2025

Uh oh!

misi1987107 commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

misi1987107 commented Apr 10, 2025 •

edited

Loading

misi1987107 commented Apr 16, 2025 •

edited

Loading