[Feature][Connector-JDBC] Fix Oracle BLOB data format preservation issue #9270

yzeng1618 · 2025-05-06T06:41:41Z

Purpose of this pull request

This PR fixes an issue with the JDBC connector where Oracle BLOB data is not properly preserved during synchronization. Currently, when transferring BLOB fields (containing text, XML, HTML, etc.) from Oracle to target systems like Doris, the data is converted to Base64-encoded strings, making it unusable in its original format.

The implementation enhances the OracleTypeConverter to properly handle BLOB data based on the handle_blob_as_string configuration parameter:

When handle_blob_as_string=true, BLOB data is treated as STRING type, preserving the original content format
When handle_blob_as_string=false (default), BLOB data is treated as BYTES type as before

Does this PR introduce any user-facing change?

Yes, this PR introduces a user-facing change in how Oracle BLOB data is handled during synchronization. Users can now configure the JDBC connector to preserve the original format of BLOB data by setting handle_blob_as_string=true in their connector configuration.

Before this change, Oracle BLOB data containing structured content like XML or HTML would be converted to Base64-encoded strings in the target system, making it difficult to use. With this change, users can choose to preserve the original content format.

How was this patch tested?

Manually tested with Oracle tables containing BLOB fields with various data types (text, XML, HTML)
Verified that the original data format is preserved when handle_blob_as_string=true

For local testing verification, for example, when the parameter handle_blob_as_string=false or not set, the situation is as follows: In the Oracle source table (TEST_BLOB_TABLE), we have BLOB data with different content types:

Row 1: Simple text "Hello, World!"

Row 2: XML content

Row 3: HTML content

However, after synchronization to the Doris target table, all BLOB data is converted to Base64-encoded strings:

Row 1: "SGVsbG8sIFdvcmxkIQ=="

Row 2: "PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4..."

Row 3: "PCFET0NUWVBFIGh0bWw+PGh0bWwgc3R5bGU9Im92..."

When the parameter is set to true, the BLOB fields in the Doris target table maintain data consistent with the source data.

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[x ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config
[x ] Update the release-note.

…o support large blob fields.

Copilot

Pull Request Overview

This pull request fixes the Oracle BLOB data preservation issue by adding a configuration parameter (handle_blob_as_string) to control whether BLOB data is converted to a string or remains as bytes. The changes span multiple modules, including source conversion methods, dialect factories, config builders, and catalog creation for both JDBC and CDC connectors.

Reviewed Changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/utils/JdbcFieldTypeUtils.java	Added special handling for BLOB conversion in getString().
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/utils/JdbcCatalogUtils.java	Configured catalog options to include handle_blob_as_string.
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/JdbcSourceFactory.java	Propagated the new configuration parameter to the dialect loader.
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/oracle/*	Updated OracleTypeMapper, OracleTypeConverter, OracleDialect, and OracleCatalog to support handle_blob_as_string.
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/config/*	Updated configuration options and builders to include handle_blob_as_string.
seatunnel-connectors-v2/connector-cdc/connector-cdc-oracle/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/oracle/utils/OracleTypeUtils.java	Adjusted conversion call for CDC connector with the new parameter.

Files not reviewed (1)

.github/actions/get-workflow-origin: Language not supported

Copilot · 2025-05-07T01:52:15Z

...rc/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/oracle/utils/OracleTypeUtils.java

@@ -69,6 +69,6 @@ public static org.apache.seatunnel.api.table.catalog.Column convertToSeaTunnelCo
            builder.scale(column.length());
        }

-        return new OracleTypeConverter(false).convert(builder.build());
+        return new OracleTypeConverter(false, false).convert(builder.build());


The CDC OracleTypeUtils conversion hardcodes handleBlobAsString to false. Consider exposing this configuration so that the CDC connector's handling of BLOB data can be aligned with the JDBC connector if needed.

Suggested change

return new OracleTypeConverter(false, false).convert(builder.build());

return new OracleTypeConverter(handleBlobAsString, false).convert(builder.build());

Hisoka-X · 2025-05-07T02:37:36Z

.github/actions/get-workflow-origin

Please do not update the version of action.

OK, I will restore the previous version.

Hisoka-X · 2025-05-07T02:44:21Z

.../apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/oracle/OracleTypeConverter.java

+                if (handleBlobAsString) {
+                    builder.dataType(BasicType.STRING_TYPE);
+                    builder.columnLength(BYTES_4GB - 1);
+                    log.info("Converted BLOB to STRING_TYPE with length: {}", BYTES_4GB - 1);
+                } else {
+                    builder.dataType(PrimitiveByteArrayType.INSTANCE);
+                    builder.columnLength(BYTES_4GB - 1);
+                    log.info(
+                            "Converted BLOB to PrimitiveByteArrayType with length: {}",
+                            BYTES_4GB - 1);
+                }


why not use CLOB to store string? Because blob was originally used to store bytes.

Thank you for your question.The design choice to handle BLOB data as strings is based on real customer use cases. We've encountered numerous situations where customers store XML files or other textual content in Oracle BLOB fields rather than CLOB, particularly in legacy systems or specific application scenarios.

When attempting to process these BLOB data directly in SQL queries using type conversions (like TO_CLOB), users often face byte size limitations, especially with large XML files. These limitations can result in data truncation or conversion failures.

This is why we implemented the handleBlobAsString option, providing users with flexibility during the ETL process:

When the data is actually textual content (like XML), it can be converted to STRING_TYPE

When the data is genuinely binary, it maintains its original binary form
This approach circumvents Oracle's internal type conversion limitations, offering more reliable processing capabilities for large textual data while maintaining backward compatibility.

make sense to me. cc @hailin0 @corgy-w

…n when building ClickHouse Nodes (apache#9277)

…o support large blob fields.

Hisoka-X · 2025-05-07T11:14:28Z

Please add test case.
Please open github action on your fork repository. Please refer https://github.com/apache/seatunnel/pull/9270/checks?check_run_id=41700794268

zengyi added 2 commits May 6, 2025 12:00

[Feature][Connector-V2] Modify the mapping method of the blob field t…

9823760

…o support large blob fields.

[Feature][Connector-V2] Modify the mapping method of the blob field t…

d9c6847

…o support large blob fields.

github-actions bot added CI&CD connectors-v2 labels May 6, 2025

[Fix][CI] Remove dead link check from sanity check (apache#9274)

0b1122c

hailin0 requested a review from Copilot May 7, 2025 01:51

Copilot AI reviewed May 7, 2025

View reviewed changes

Hisoka-X added the First-time contributor First-time contributor label May 7, 2025

Hisoka-X linked an issue May 7, 2025 that may be closed by this pull request

[Feature][connector-jdbc] Fix Oracle BLOB data corruption in JDBC connector #9268

Open

3 tasks

Hisoka-X reviewed May 7, 2025

View reviewed changes

mrtisttt and others added 4 commits May 7, 2025 11:50

[Fix][Connector-V2] Fix the problem that missing options configuratio…

051d19c

…n when building ClickHouse Nodes (apache#9277)

[Feature][Connector-V2] Modify the mapping method of the blob field t…

2410560

…o support large blob fields.

[Feature][Connector-V2] Modify the mapping method of the blob field t…

7793f74

…o support large blob fields.

Merge remote-tracking branch 'origin/BDPL-30699' into BDPL-30699

9163314

[Feature][connector-jdbc] add unit test and remove unnecessary comments.

bfc2008

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Connector-JDBC] Fix Oracle BLOB data format preservation issue #9270

[Feature][Connector-JDBC] Fix Oracle BLOB data format preservation issue #9270

yzeng1618 commented May 6, 2025

Copilot AI left a comment

Copilot AI May 7, 2025

Hisoka-X May 7, 2025

yzeng1618 May 7, 2025

Hisoka-X May 7, 2025

yzeng1618 May 7, 2025

Hisoka-X May 7, 2025

Hisoka-X commented May 7, 2025

	return new OracleTypeConverter(false, false).convert(builder.build());
	return new OracleTypeConverter(handleBlobAsString, false).convert(builder.build());

[Feature][Connector-JDBC] Fix Oracle BLOB data format preservation issue #9270

Are you sure you want to change the base?

[Feature][Connector-JDBC] Fix Oracle BLOB data format preservation issue #9270

Conversation

yzeng1618 commented May 6, 2025

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Copilot AI May 7, 2025

Choose a reason for hiding this comment

Hisoka-X May 7, 2025

Choose a reason for hiding this comment

yzeng1618 May 7, 2025

Choose a reason for hiding this comment

Hisoka-X May 7, 2025

Choose a reason for hiding this comment

yzeng1618 May 7, 2025

Choose a reason for hiding this comment

Hisoka-X May 7, 2025

Choose a reason for hiding this comment

Hisoka-X commented May 7, 2025