how to replicate mediumblob fields #55

pablotrin · 2022-01-12T15:53:29Z

pablotrin
Jan 12, 2022

My source mysql database have a couple of mediumblob fileds. How can replicate them to my sink (also Mysql) database?
As I'm getting this error:

2022-01-12 15:30:36,940 ERROR ReplicaTask:74 ERROR in TaskId-0 inserting data to sink table: (conn=77) Invalid utf8 character string: 'GSI���n%G"/�eHC�9����TQ�sC�����p?'
2022-01-12 15:30:36,941 ERROR ReplicaDB:136 Got exception running ReplicaDB:
java.util.concurrent.ExecutionException: java.sql.SQLTransientConnectionException: (conn=77) Invalid utf8 character string: 'GSI���n%G"/�eHC�9����TQ�sC�����p?'
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_212]
        at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_212]
        at org.replicadb.ReplicaDB.processReplica(ReplicaDB.java:117) [ReplicaDB-0.11.4.jar:0.11.4]
        at org.replicadb.ReplicaDB.main(ReplicaDB.java:47) [ReplicaDB-0.11.4.jar:0.11.4]
Caused by: java.sql.SQLTransientConnectionException: (conn=77) Invalid utf8 character string: 'GSI���n%G"/�eHC�9����TQ�sC�����p?'
        at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:79) ~[mariadb-java-client-2.7.3.jar:?]
        at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:155) ~[mariadb-java-client-2.7.3.jar:?]
        at org.mariadb.jdbc.MariaDbStatement.executeExceptionEpilogue(MariaDbStatement.java:274) ~[mariadb-java-client-2.7.3.jar:?]
        at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:363) ~[mariadb-java-client-2.7.3.jar:?]
        at org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:627) ~[mariadb-java-client-2.7.3.jar:?]
        at org.replicadb.manager.MySQLManager.insertDataToTable(MySQLManager.java:154) ~[ReplicaDB-0.11.4.jar:0.11.4]
        at org.replicadb.ReplicaTask.call(ReplicaTask.java:70) ~[ReplicaDB-0.11.4.jar:0.11.4]
        at org.replicadb.ReplicaTask.call(ReplicaTask.java:14) ~[ReplicaDB-0.11.4.jar:0.11.4]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212]
Caused by: org.mariadb.jdbc.internal.util.exceptions.MariaDbSqlException: Invalid utf8 character string: 'GSI���n%G"/�eHC�9����TQ�sC�����p?'
        at org.mariadb.jdbc.internal.util.exceptions.MariaDbSqlException.of(MariaDbSqlException.java:34) ~[mariadb-java-client-2.7.3.jar:?]
        at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:194) ~[mariadb-java-client-2.7.3.jar:?]
        at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:262) ~[mariadb-java-client-2.7.3.jar:?]
        at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:357) ~[mariadb-java-client-2.7.3.jar:?]
        at org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:627) ~[mariadb-java-client-2.7.3.jar:?]
        at org.replicadb.manager.MySQLManager.insertDataToTable(MySQLManager.java:154) ~[ReplicaDB-0.11.4.jar:0.11.4]
        at org.replicadb.ReplicaTask.call(ReplicaTask.java:70) ~[ReplicaDB-0.11.4.jar:0.11.4]
        at org.replicadb.ReplicaTask.call(ReplicaTask.java:14) ~[ReplicaDB-0.11.4.jar:0.11.4]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212]
Caused by: java.sql.SQLException: Invalid utf8 character string: 'GSI���n%G"/�eHC�9����TQ�sC�����p?'
        at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:1694) ~[mariadb-java-client-2.7.3.jar:?]
        at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1556) ~[mariadb-java-client-2.7.3.jar:?]
        at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1519) ~[mariadb-java-client-2.7.3.jar:?]
        at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readLocalInfilePacket(AbstractQueryProtocol.java:1798) ~[mariadb-java-client-2.7.3.jar:?]
        at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1562) ~[mariadb-java-client-2.7.3.jar:?]
        at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1519) ~[mariadb-java-client-2.7.3.jar:?]
        at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:256) ~[mariadb-java-client-2.7.3.jar:?]
        at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:357) ~[mariadb-java-client-2.7.3.jar:?]
        at org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:627) ~[mariadb-java-client-2.7.3.jar:?]
        at org.replicadb.manager.MySQLManager.insertDataToTable(MySQLManager.java:154) ~[ReplicaDB-0.11.4.jar:0.11.4]
        at org.replicadb.ReplicaTask.call(ReplicaTask.java:70) ~[ReplicaDB-0.11.4.jar:0.11.4]
        at org.replicadb.ReplicaTask.call(ReplicaTask.java:14) ~[ReplicaDB-0.11.4.jar:0.11.4]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212]

osalvador · 2022-01-21T09:02:01Z

osalvador
Jan 21, 2022
Maintainer

Hi @pablotrin ,

Maybe there is a bug, could you share the DDL of the source and sink tables and the replicadb configuration file?

Thanks.

0 replies

pablotrin · 2022-01-21T11:24:04Z

pablotrin
Jan 21, 2022
Author

Hi, Here are the tables and config file. Thanks. source table: ---------------- CREATE TABLE `protocolos` ( `id_empresa` VARCHAR(20) NOT NULL COLLATE 'utf8_general_ci', `id_orden` INT(11) NOT NULL, `fecha_orden` DATETIME NULL DEFAULT NULL, `nombre_apellido_paciente` VARCHAR(60) NULL DEFAULT NULL COLLATE 'utf8_general_ci', `tipo_doc_paciente` VARCHAR(3) NULL DEFAULT NULL COLLATE 'utf8_general_ci', `nro_doc_paciente` VARCHAR(40) NULL DEFAULT NULL COLLATE 'utf8_general_ci', `matricula_profesional` VARCHAR(15) NULL DEFAULT NULL COLLATE 'utf8_general_ci', `diagnostico` VARCHAR(250) NULL DEFAULT NULL COLLATE 'utf8_general_ci', `protocolo_PDF` MEDIUMBLOB NULL DEFAULT NULL, `fecha_nac_paciente` DATE NULL DEFAULT NULL, `sexo_paciente` VARCHAR(1) NULL DEFAULT NULL COLLATE 'utf8_general_ci', `id_laboratorio_derivante` INT(11) NULL DEFAULT NULL, `pdf_confidencial` MEDIUMBLOB NULL DEFAULT NULL, `centro` VARCHAR(50) NOT NULL DEFAULT '' COLLATE 'utf8_general_ci', PRIMARY KEY (`id_empresa`, `id_orden`, `centro`) USING BTREE, INDEX `id_empresa` (`id_empresa`) USING BTREE, INDEX `id_orden` (`id_orden`) USING BTREE, INDEX `protocolos_tipo_doc_paciente_IDX` (`tipo_doc_paciente`, `nro_doc_paciente`) USING BTREE ) COLLATE='utf8_general_ci' ENGINE=InnoDB ; sink table: ------------- CREATE TABLE `protocolos` ( `id_empresa` VARCHAR(20) NOT NULL DEFAULT '0' COLLATE 'utf8_general_ci', `id_orden` INT(10) NOT NULL DEFAULT '0', `fecha_orden` DATETIME NULL DEFAULT NULL, `nombre_apellido_paciente` VARCHAR(60) NULL DEFAULT NULL COLLATE 'utf8_general_ci', `tipo_doc_paciente` VARCHAR(3) NULL DEFAULT NULL COLLATE 'utf8_general_ci', `nro_doc_paciente` VARCHAR(40) NULL DEFAULT NULL COLLATE 'utf8_general_ci', `matricula_profesional` VARCHAR(15) NULL DEFAULT NULL COLLATE 'utf8_general_ci', `diagnostico` VARCHAR(250) NULL DEFAULT NULL COLLATE 'utf8_general_ci', `protocolo_PDF` MEDIUMBLOB NULL DEFAULT NULL, `fecha_nac_paciente` DATE NULL DEFAULT NULL, `sexo_paciente` VARCHAR(1) NULL DEFAULT NULL COLLATE 'utf8_general_ci', `id_laboratorio_derivante` INT(10) NULL DEFAULT NULL, `pdf_confidencial` MEDIUMBLOB NULL DEFAULT NULL, `centro` VARCHAR(50) NOT NULL DEFAULT '' COLLATE 'utf8_general_ci', PRIMARY KEY (`id_empresa`, `id_orden`, `centro`) USING BTREE, INDEX `id_empresa` (`id_empresa`) USING BTREE, INDEX `id_orden` (`id_orden`) USING BTREE, INDEX `protocolos_tipo_doc_paciente_IDX` (`tipo_doc_paciente`, `nro_doc_paciente`) USING BTREE ) COLLATE='utf8_general_ci' ENGINE=InnoDB ; config file ------------- ######################## ReplicadB General Options ######################## mode=incremental jobs=1 fetch.size=100 verbose=false ## Help us to improve by activating usage analysis through the Sentry service. Or set up your own Sentry DSN sentry.dsn= ***@***.***/5962725 ############################# Soruce Options ############################## source.connect=jdbc:mysql://xxxxxxxxxx:8360/ zzzzzzzzz source.user=<user> source.password=<pass> source.table=protocolos source.where=id_empresa = '123456' ############################# Sink Options ################################ sink.connect=jdbc:mysql://yyyyyyyyyy:3306/zzzzzzzzz sink.user=<other_user> sink.password=<other_pass> sink.table=protocolos sink.staging.table=protocolos_1 El vie, 21 ene 2022 a la(s) 06:02, Oscar Salvador Magallanes ( ***@***.***) escribió:

…

Hi @pablotrin <https://github.com/pablotrin> , Maybe there is a bug, could you share the DDL of the source and sink tables and the replicadb configuration file? Thanks. — Reply to this email directly, view it on GitHub <#55 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGO3Q5UNEML43MEC6U6O4DUXEOJNANCNFSM5LZKUC7Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

0 replies

osalvador · 2022-02-07T12:01:37Z

osalvador
Feb 7, 2022
Maintainer

Hi @pablotrin

There is a new version v0.11.8 that fixes the handling of binary data in MySQL and MariaDB.

Does this solve your bug?

Regards!

3 replies

pablotrin Feb 8, 2022
Author

Thanks for taking the time to work on this.
Unfortunately it still does not work, although data is copied, but apparently it gets "converted" to a different format, as the binary of a PDF stored on a field cannot be read from the replicated database, and when looking at the field in the database, the data has changed.

osalvador Mar 1, 2022
Maintainer

Hi @pablotrin

I've created an issue #64, I'll solve it as soon as possible.

Thank you!

pablotrin Mar 4, 2022
Author

thank you!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

how to replicate mediumblob fields #55

Uh oh!

{{title}}

Uh oh!

Replies: 3 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

how to replicate mediumblob fields #55

Uh oh!

pablotrin Jan 12, 2022

Replies: 3 comments · 3 replies

Uh oh!

osalvador Jan 21, 2022 Maintainer

Uh oh!

pablotrin Jan 21, 2022 Author

Uh oh!

osalvador Feb 7, 2022 Maintainer

Uh oh!

pablotrin Feb 8, 2022 Author

Uh oh!

Uh oh!

osalvador Mar 1, 2022 Maintainer

Uh oh!

pablotrin Mar 4, 2022 Author

pablotrin
Jan 12, 2022

Replies: 3 comments 3 replies

osalvador
Jan 21, 2022
Maintainer

pablotrin
Jan 21, 2022
Author

osalvador
Feb 7, 2022
Maintainer

pablotrin Feb 8, 2022
Author

osalvador Mar 1, 2022
Maintainer

pablotrin Mar 4, 2022
Author