Debezium MySQL Connector stopped streaming CDC data — how to safely resume from binlog?

[amiteka1300]

Hello team,

We have been running a Debezium MySQL connector on Strimzi for a few years without issues. Suddenly, the connector stopped streaming data even though the status API shows it as RUNNING and there are no errors in the logs.

---

Original Connector (Example)

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: mysql-connector-orders
  namespace: kafka-prod
  labels:
    strimzi.io/cluster: kafka-connect-prod
spec:
  class: io.debezium.connector.mysql.MySqlConnector
  tasksMax: 1
  config:
    tasks.max: 1
    database.hostname: orders-db.prod.internal
    database.port: 3306
    database.user: debezium_user
    database.password: ${secrets:kafka-prod/mysql-credentials:password}
    database.server.id: 10001
    topic.prefix: mysql
    database.include.list: orders_db
    table.include.list: orders_db.*
    schema.history.internal.kafka.bootstrap.servers: kafka-bootstrap:9092
    schema.history.internal.kafka.topic: schema-changes.mysql.orders
    snapshot.mode: schema_only
    snapshot.locking.mode: none
    value.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter.schemas.enable: false
    key.converter: org.apache.kafka.connect.json.JsonConverter
    key.converter.schemas.enable: false

This connector status keeps showing RUNNING with HTTP 200, but no CDC data is produced.

Recovery Attempt (Example)

We created a new connector with a new name and server ID and tried to start from the last known binlog position:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: mysql-connector-orders-recovery
  namespace: kafka-prod
  labels:
    strimzi.io/cluster: kafka-connect-prod
spec:
  class: io.debezium.connector.mysql.MySqlConnector
  tasksMax: 1
  config:
    tasks.max: 1
    database.hostname: orders-db.prod.internal
    database.port: 3306
    database.user: debezium_user
    database.password: ${secrets:kafka-prod/mysql-credentials:password}
    database.server.id: 10002
    topic.prefix: mysql
    database.include.list: orders_db
    table.include.list: orders_db.*
    schema.history.internal.kafka.bootstrap.servers: kafka-bootstrap:9092
    schema.history.internal.kafka.topic: schema-changes.mysql.orders
    database.initial.binlog.filename: mysql-bin-changelog.012345
    database.initial.binlog.position: 1234567
    snapshot.mode: schema_only
    snapshot.locking.mode: none
    value.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter.schemas.enable: false
    key.converter: org.apache.kafka.connect.json.JsonConverter
    key.converter.schemas.enable: false


But even with the binlog filename and position, it does not resume correctly.

Questions

What is the proper way to recover from this without losing data?

Is there a way to manually reset connector offsets to a specific binlog position in Strimzi-managed Kafka Connect?

Any best practices for handling a connector that is RUNNING but stopped producing events?

Environment

Mysql connector plugin verion: 2.7.1.Final

Kubernetes: 1.31

[scholzj]

Restarting connectors and reseting offsets is described in our docs: https://strimzi.io/docs/operators/latest/full/deploying.html#con-kafka-connector-config-str. The rest is likely Debezium specific, so it would be best to ask in some Debezium forum.

[amiteka1300]

hi @scholzj ,
I had tried with restarting the connectors but that also did not worked. When i created new connectors with new server id and connector name then it started working. i have also opened ticket on Debezium community.