Qlick Replicate other files formats

[jrevillard]

Hello,

I would like to try to play with you nice connector but my use case is a bit different in a way that Qlik Replicate store the data directly into hdfs in sequence file format. I'm able to put those sequence files inside kafka using the https://www.confluent.io/hub/mmolimar/kafka-connect-fs connector but of course... this is not JSON.

Would it be a lot of work to adapt what you did ?

Best.

[danielpetisme]

Hi.

Could you elaborate, please. I'm not sure to understand your usecase.
What's your data source, a Database, files, other?
What's your target data repository, Hdfs, Kafka, other?

Here you have the story behinds the repo
https://blogit.michelin.io/open-sourcing-kafka-connect-transforms-for-qlik-replicate/

We configured Qlik Replicate to read database events (ie. change data capture events) and replicate them to a Kafka cluster.
Then we have been configuration a Kafka Connect JDBC Sink Connector to read these CDC events and persists them into another database.
Because the CDC event structure wasn't compatible "as-is" with the JDBC connector, we introduced a SMT (ie. Single Message Transformation) to transform the CDC event into a structure compatible with JDBC connector.

[jrevillard]

My use case is approximately the same. I want to be able to use the jdbc connector to reconstruct the tables using the data provided by Qlik Replicate. The difference for me is that Qlik do not send the data in Kafka but directly in hdfs in sequence file format: 1 directory for initial load and another one for changes. I'm able to send this data back in Kafka and I'm pretty sure that the structure would be nearly the same... But not in JSON. So I wonder how we could benefit from what you did in our use case.

I can show you some sample data if you want.

Best,
Jérôme

[danielpetisme]

My 2cts, it would be easier to either send the DB events to both Hadoop and Kafka in parallel or send it first to Kafka and then use the HdfsSink Connector for Hdfs and JdbcSink Connector for your target DB. As a rule of thumb, is harder to extract data at rest than it is to persist data in motion 😉 .

Now, if you can't reconsider your architecture, please share sample data to see how different the structures are.

[jrevillard]

Thanks @danielpetisme ... sorry, I'm a bit overloaded actually so I'm not really responsive :-)

I agree with your remark. The only think that is important for me is not to overload the Qlick Replicate server. Honestly I did not test yet, but as we have some tables with billions of row and millions of modifications each days ans as it is perfectly working actually, I wanted to make something as less intrusive as possible for the actual workflow... but I could reconsider it of course if it's really too complicated.

As soon as I have a bit of time, I will share with you some sample data.

Best,
Jerome