[Feature] Skywalking to monitor change data capture application connector endpoints

[ccfahe]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

I have a change dta capture service which uses Debezium(kafak-zookeeper and connectors ) for gettiong updates from Oracle and updating them to different DB (postgrteSql).I tried to serach any feature that can help to monitor the endpoints that is connectors(oracle-debezium) and connector(Debezium-postgresql)

I am using ojdbc connector for oracle

sample link to understand CDC with Oracle 11g version(currently uisng 19c)
https://muirandy.wordpress.com/2019/11/06/debezium-with-oracle-11g/

Use case

CDC

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

No, there is no such thing.

[wu-sheng]

Oracle is commercial prod, it is out of open source hands.

[ccfahe]

I agree,but it need to monitor the flow of data in debezium(flow from oracle to connector and outside the connector) irrespective of DB

[ccfahe]

I will ask other way: do we have support for debezium service

[wu-sheng]

AFAIK, no. Tracing is tracking the execution flow, usually not observe all data flow. The ETL and data transfer pipeline is rarely to be traced, considering it is data oriented.

[ccfahe]

Hi @wu-sheng ,

First of all thanks for taking time in replying my queries.I really appreciate it :)

I believe that you have misunderstood the question, so let me rephrase my question.
 
The basic flow of CDC is
 
Oracle transaction log read by Debezium -> Data recorded in Kafka (using debezium api) Data read from Kafka (using debezium api)-> Data inserted in PostgreSQL (using jdbc queries)
We need Apache Skywalking tool to monitor the latency at the api calls during data insertion in Kafka by Debezium API and subsequent reads from Kafka by Debezium API and insertion in PostgreSQL. Is it possible to do so?
 
Also, I want to understand in above flow (CDC) how Apache SkyWalking can help us with its features. I have noted down the features such as detecting the slow queries in database and latency detection in Kafka data flow. Is there any other feature available for Debezium, Kafka services, if yes kindly suggest .

[wu-sheng]

Then you need agent running at both side, if the header is well set, especially sw-x(Jave agent implemented, it is optional). You could have MQ consuming latency.

We have plugin for Kafka in Java. But I don't know anyone used in a stack like yours. So, you have to try on your own.