Kafka connect exposing jmx metrics

[hafizmujadidKhalid]

Hello lovely Strimzi community!
I have deployed kafka connect using strimzi and that works undoubtedly. Now, I am trying to enable monitoring for kafka connect cluster. I read jmxOptions can be enabled it. I did it but I can't access any metric on the 9999 port of the kafka-connect pod.
Here is my yaml configuration:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: ingestion-metric
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  replicas: 1
  version: 2.6.0
  image: <ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com/kafka-connect:2.1.0
  bootstrapServers: kafka-broker:12345
  metricsConfig:
    type: jmxPrometheusExporter
    valueFrom:
      configMapKeyRef:
        name: ingestion-metric
        key: ingestion-metric-config.yml
  resources:
    requests:
      cpu: 250m
      memory: 512Mi
    limits:
      cpu: 500m
      memory: 1Gi
  jmxOptions: {}
  jvmOptions:
    gcLoggingEnabled: false
  config:
    group.id: ingestion-metric

    key.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter: org.apache.kafka.connect.json.JsonConverter
    offset.storage.topic: ingestion-metric-offsets
    config.storage.topic: ingestion-metric-configs
    status.storage.topic: ingestion-metric-status
    config.storage.replication.factor: 3
    offset.storage.replication.factor: 3
    status.storage.replication.factor: 3
  logging:
    type: inline
    loggers:
      logger.name: "DEBUG"

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: ingestion-metric
  labels:
    app: strimzi
data:
  ingestion-metric-config.yml: |
    lowercaseOutputName: true
    lowercaseOutputLabelNames: true
    rules:
    #kafka.connect:type=connect-node-metrics,client-id="{clientid}",node-id="{nodeid}"
    #kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id="{nodeid}"
    - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), node-id=(.+)><>(.+-total|.+-avg)
      name: kafka_$2_$5
      labels:
        clientId: "$3"
        nodeId: "$4"
      help: "Kafka $1 JMX metric type $2"
      type: UNTYPED

    #kafka.connect:type=kafka-metrics-count,client-id="{clientid}"
    #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}"
    #kafka.consumer:type=consumer-coordinator-metrics,client-id="{clientid}"
    #kafka.consumer:type=consumer-metrics,client-id="{clientid}"
    - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.*)><>(.+-total|.+-avg|.+-bytes|.+-count|.+-ratio|.+-age|.+-flight|.+-threads|.+-connectors|.+-tasks|.+-ago)
      name: kafka_$2_$4
      labels:
        clientId: "$3"
      help: "Kafka $1 JMX metric type $2"
      type: GAUGE

    #kafka.connect:type=connector-metrics,connector="{connector}"
    - pattern: 'kafka.(.+)<type=connector-metrics, connector=(.+)><>(connector-class|connector-type|connector-version|status): (.+)'
      name: kafka_connect_connector_$3
      value: 1
      labels:
        connector: "$2"
        $3: "$4"
      help: "Kafka Connect $3 JMX metric type connector"
      type: GAUGE

    #kafka.connect:type=connector-task-metrics,connector="{connector}",task="{task}<> status"
    - pattern: 'kafka.connect<type=connector-task-metrics, connector=(.+), task=(.+)><>status: ([a-z-]+)'
      name: kafka_connect_connector_task_status
      value: 1
      labels:
        connector: "$1"
        task: "$2"
        status: "$3"
      help: "Kafka Connect JMX Connector task status"
      type: GAUGE

    #kafka.connect:type=task-error-metrics,connector="{connector}",task="{task}"
    #kafka.connect:type=source-task-metrics,connector="{connector}",task="{task}"
    #kafka.connect:type=sink-task-metrics,connector="{connector}",task="{task}"
    #kafka.connect:type=connector-task-metrics,connector="{connector}",task="{task}"
    - pattern: kafka.connect<type=(.+)-metrics, connector=(.+), task=(.+)><>(.+-total|.+-count|.+-ms|.+-ratio|.+-seq-no|.+-rate|.+-max|.+-avg|.+-failures|.+-requests|.+-timestamp|.+-logged|.+-errors|.+-retries|.+-skipped)
      name: kafka_connect_$1_$4
      labels:
        connector: "$2"
        task: "$3"
      help: "Kafka Connect JMX metric type $1"
      type: GAUGE

    #kafka.connect:type=connector-metrics,connector="{connector}"
    #kafka.connect:type=connect-worker-metrics,connector="{connector}"
    - pattern: kafka.connect<type=connect-worker-metrics, connector=(.+)><>([a-z-]+)
      name: kafka_connect_worker_$2
      labels:
        connector: "$1"
      help: "Kafka Connect JMX metric $1"
      type: GAUGE

    #kafka.connect:type=connect-worker-metrics
    - pattern: kafka.connect<type=connect-worker-metrics><>([a-z-]+)
      name: kafka_connect_worker_$1
      help: "Kafka Connect JMX metric worker"
      type: GAUGE

    #kafka.connect:type=connect-worker-rebalance-metrics
    - pattern: kafka.connect<type=connect-worker-rebalance-metrics><>([a-z-]+)
      name: kafka_connect_worker_rebalance_$1
      help: "Kafka Connect JMX metric rebalance information"
      type: GAUGE

The pod looks fine from strimzi operator logs and kafka connect logs. But still I can't see any metrics. I entered into kafkaconnect pod and executed the curl localhost:9999/metrics command. It says connection refused. I don't have any Prometheus yet. Am I missing something? Do I need to install something else? Thanks

[scholzj]

I think you are mixing two different things:

• jmxOptions configures JMX access on port 9999. If you enable it, you can access the JMX MBeans using VisualVM or Jconsole.
• metricsConfig enables Prometheus metrics on port 9404. That is what you can scrape using Prometheus or using curl (but on port 9404).

So you would normally pick one. If you want metrics, we recommend Prometheus over JMX -> so you can remove the jmxOptions, keep the metricsConfig and get the metrics on 9404.

[hafizmujadidKhalid]

So, metrics are available inside the pod on port 9404 and i should use prometheous scraper to scrape pod for metrics? Out of curiosity, Even though, I enabled both, shouldn't jmx metrics be available on 9999 port of service? thanks already for quick response.

[scholzj]

The JMX 9999 port is IMHO added to the service (well, I did not tried it right now as I do not use JMX - but the code say it should be added). Although I think it is not really too useful if you have more Connect nodes because the service will lead you to one random pod basically.

The 9404 port for the Prometheus metrics is not exposed on the service.

[hafizmujadidKhalid]

Thanks, I have checked service definition and jmx port 9999 was leading me to access jmx mbeans on 9999.

[scholzj]

Yes, it would give you the JMX access (i.e. it matches the jmxOptions).

[ChinthapalliNikhithaChandana]

Hello, I did enable JMX exporter using metricsConfig in kafkaConnect, and that enabled port 9404 on my pod.
Next, I created a podMonitor to scrape those metrics and send to Prometheus. But I'm seeing a connection refused error.

I'm using Strimzi 0.48.0

[ChinthapalliNikhithaChandana]

Hello,

kubectl get kafkaconnect rig-connect -n rigstream -o yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: rigstream-kafka-connect:kafka.strimzi.io/KafkaConnect:rigstream/rig-connect
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"kafka.strimzi.io/v1beta2","kind":"KafkaConnect","metadata":{"annotations":{"argocd.argoproj.io/tracking-id":"rigstream-kafka-connect:kafka.strimzi.io/KafkaConnect:rigstream/rig-connect","strimzi.io/use-connector-resources":"true"},"labels":{"app.kubernetes.io/instance":"rigstream-kafka-connect","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"rigstream-kafka-connect","app.kubernetes.io/version":"4.0.0","helm.sh/chart":"rigstream-kafka-connect-0.1.0-dev.20260120.2"},"name":"rig-connect","namespace":"rigstream"},"spec":{"bootstrapServers":"rig-kafka-kafka-bootstrap.rigstream:9092","config":{"config.storage.replication.factor":-1,"config.storage.topic":"rig-connect-configs","consumer.fetch.max.wait.ms":500,"consumer.fetch.min.bytes":1048576,"consumer.max.poll.interval.ms":600000,"consumer.max.poll.records":5000,"group.id":"rig-connect","key.converter":"org.apache.kafka.connect.storage.StringConverter","offset.storage.replication.factor":-1,"offset.storage.topic":"rig-connect-offsets","status.storage.replication.factor":-1,"status.storage.topic":"rig-connect-status","value.converter":"io.confluent.connect.avro.AvroConverter","value.converter.schema.registry.url":"http://rigstream-schema-registry.rigstream:8081"},"externalConfiguration":{"env":[{"name":"DB_PASSWORD","valueFrom":{"secretKeyRef":{"key":"password","name":"rig-postgres-secret"}}}]},"image":"harbor.sperrydigital.local/righive/kafka-connect-rig-sink:latest","metricsConfig":{"type":"jmxPrometheusExporter","valueFrom":{"configMapKeyRef":{"key":"kafka-connect-metrics-config.yaml","name":"kafka-connect-metrics"}}},"replicas":1,"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"1","memory":"2Gi"}},"template":{"pod":{"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"kubernetes.io/os","operator":"In","values":["linux"]}]}]}},"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"podAffinityTerm":{"labelSelector":{"matchLabels":{"strimzi.io/name":"rig-connect"}},"topologyKey":"kubernetes.io/hostname"},"weight":100}]}},"terminationGracePeriodSeconds":60}},"version":"4.0.0"}}
    strimzi.io/use-connector-resources: "true"
  creationTimestamp: "2026-01-20T20:40:37Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: rigstream-kafka-connect
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: rigstream-kafka-connect
    app.kubernetes.io/version: 4.0.0
    helm.sh/chart: rigstream-kafka-connect-0.1.0-dev.20260120.2
  name: rig-connect
  namespace: rigstream
  resourceVersion: "31604077"
  uid: aa6e3ce7-842a-42e8-b5f2-f93d48fcc58b
spec:
  bootstrapServers: rig-kafka-kafka-bootstrap.rigstream:9092
  config:
    config.storage.replication.factor: -1
    config.storage.topic: rig-connect-configs
    consumer.fetch.max.wait.ms: 500
    consumer.fetch.min.bytes: 1048576
    consumer.max.poll.interval.ms: 600000
    consumer.max.poll.records: 5000
    group.id: rig-connect
    key.converter: org.apache.kafka.connect.storage.StringConverter
    offset.storage.replication.factor: -1
    offset.storage.topic: rig-connect-offsets
    status.storage.replication.factor: -1
    status.storage.topic: rig-connect-status
    value.converter: io.confluent.connect.avro.AvroConverter
    value.converter.schema.registry.url: http://rigstream-schema-registry.rigstream:8081
  externalConfiguration:
    env:
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          key: password
          name: rig-postgres-secret
  image: harbor.sperrydigital.local/righive/kafka-connect-rig-sink:latest
  metricsConfig:
    type: jmxPrometheusExporter
    valueFrom:
      configMapKeyRef:
        key: kafka-connect-metrics-config.yaml
        name: kafka-connect-metrics
  replicas: 1
  resources:
    limits:
      cpu: "2"
      memory: 4Gi
    requests:
      cpu: "1"
      memory: 2Gi
  template:
    pod:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  strimzi.io/name: rig-connect
              topologyKey: kubernetes.io/hostname
            weight: 100
      terminationGracePeriodSeconds: 60
  version: 4.0.0
status:
  conditions:
  - lastTransitionTime: "2026-01-20T20:44:13.245555407Z"
    status: "True"
    type: Ready
  - lastTransitionTime: "2026-01-20T20:43:05.459547344Z"
    message: In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2
      the externalConfiguration property at path spec.externalConfiguration has been
      deprecated. The external configuration is deprecated and will be removed in
      the future. Please use the template section instead to configure additional
      environment variables or volumes.
    reason: DeprecatedFields
    status: "True"
    type: Warning
  - lastTransitionTime: "2026-01-20T20:43:05.459547344Z"
    message: In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2
      the object externalConfiguration at path spec.externalConfiguration has been
      deprecated. This object has been replaced with KafkaConnectTemplate.
    reason: DeprecatedObjects
    status: "True"
    type: Warning
  - lastTransitionTime: "2026-01-20T20:43:05.459547344Z"
    message: In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2
      the env property at path spec.externalConfiguration.env has been deprecated.
      The external configuration environment variables are deprecated and will be
      removed in the future. Please use the environment variables in a container template
      instead.
    reason: DeprecatedFields
    status: "True"
    type: Warning
  connectorPlugins:
  - class: io.confluent.connect.jdbc.JdbcSinkConnector
    type: sink
    version: 10.8.4
  - class: io.confluent.connect.jdbc.JdbcSourceConnector
    type: source
    version: 10.8.4
  - class: org.apache.kafka.connect.mirror.MirrorCheckpointConnector
    type: source
    version: 4.0.0
  - class: org.apache.kafka.connect.mirror.MirrorHeartbeatConnector
    type: source
    version: 4.0.0
  - class: org.apache.kafka.connect.mirror.MirrorSourceConnector
    type: source
    version: 4.0.0
  labelSelector: strimzi.io/cluster=rig-connect,strimzi.io/name=rig-connect-connect,strimzi.io/kind=KafkaConnect
  observedGeneration: 1
  replicas: 1
  url: http://rig-connect-connect-api.rigstream.svc:8083

Cluster operator pod logs

2026-01-21 15:23:10 WARN  StatusUtils:106 - Reconciliation #3265(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the externalConfiguration property at path spec.externalConfiguration has been deprecated. The external configuration is deprecated and will be removed in the future. Please use the template section instead to configure additional environment variables or volumes.
2026-01-21 15:23:10 WARN  StatusUtils:131 - Reconciliation #3265(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the object externalConfiguration at path spec.externalConfiguration has been deprecated. This object has been replaced with KafkaConnectTemplate.
2026-01-21 15:23:10 WARN  StatusUtils:106 - Reconciliation #3265(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the env property at path spec.externalConfiguration.env has been deprecated. The external configuration environment variables are deprecated and will be removed in the future. Please use the environment variables in a container template instead.
2026-01-21 15:23:10 INFO  AbstractOperator:265 - Reconciliation #3265(timer) KafkaConnect(rigstream/rig-connect): KafkaConnect rig-connect will be checked for creation or modification
2026-01-21 15:23:11 INFO  KafkaConnectAssemblyOperator:465 - Reconciliation #3265(timer) KafkaConnect(rigstream/rig-connect): creating/updating connector: rig-jdbc-sink
2026-01-21 15:23:11 INFO  KafkaConnectAssemblyOperator:465 - Reconciliation #3265(timer) KafkaConnect(rigstream/rig-connect): creating/updating connector: rig-tool-memory-jdbc-sink
2026-01-21 15:23:11 INFO  AbstractOperator:545 - Reconciliation #3265(timer) KafkaConnect(rigstream/rig-connect): reconciled
2026-01-21 15:23:12 INFO  AbstractOperator:545 - Reconciliation #3264(timer) Kafka(rigstream/rig-kafka): reconciled
2026-01-21 15:25:10 INFO  ClusterOperator:135 - Triggering periodic reconciliation for namespace rigstream
2026-01-21 15:25:10 INFO  AbstractOperator:265 - Reconciliation #3266(timer) Kafka(rigstream/rig-kafka): Kafka rig-kafka will be checked for creation or modification
2026-01-21 15:25:10 WARN  StatusUtils:106 - Reconciliation #3267(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the externalConfiguration property at path spec.externalConfiguration has been deprecated. The external configuration is deprecated and will be removed in the future. Please use the template section instead to configure additional environment variables or volumes.
2026-01-21 15:25:10 WARN  StatusUtils:131 - Reconciliation #3267(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the object externalConfiguration at path spec.externalConfiguration has been deprecated. This object has been replaced with KafkaConnectTemplate.
2026-01-21 15:25:10 WARN  StatusUtils:106 - Reconciliation #3267(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the env property at path spec.externalConfiguration.env has been deprecated. The external configuration environment variables are deprecated and will be removed in the future. Please use the environment variables in a container template instead.
2026-01-21 15:25:10 WARN  StatusUtils:106 - Reconciliation #3267(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the externalConfiguration property at path spec.externalConfiguration has been deprecated. The external configuration is deprecated and will be removed in the future. Please use the template section instead to configure additional environment variables or volumes.
2026-01-21 15:25:10 WARN  StatusUtils:131 - Reconciliation #3267(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the object externalConfiguration at path spec.externalConfiguration has been deprecated. This object has been replaced with KafkaConnectTemplate.
2026-01-21 15:25:10 WARN  StatusUtils:106 - Reconciliation #3267(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the env property at path spec.externalConfiguration.env has been deprecated. The external configuration environment variables are deprecated and will be removed in the future. Please use the environment variables in a container template instead.
2026-01-21 15:25:10 INFO  AbstractOperator:265 - Reconciliation #3267(timer) KafkaConnect(rigstream/rig-connect): KafkaConnect rig-connect will be checked for creation or modification
2026-01-21 15:25:11 INFO  KafkaConnectAssemblyOperator:465 - Reconciliation #3267(timer) KafkaConnect(rigstream/rig-connect): creating/updating connector: rig-jdbc-sink
2026-01-21 15:25:11 INFO  KafkaConnectAssemblyOperator:465 - Reconciliation #3267(timer) KafkaConnect(rigstream/rig-connect): creating/updating connector: rig-tool-memory-jdbc-sink
2026-01-21 15:25:11 INFO  AbstractOperator:545 - Reconciliation #3267(timer) KafkaConnect(rigstream/rig-connect): reconciled
2026-01-21 15:25:12 INFO  AbstractOperator:545 - Reconciliation #3266(timer) Kafka(rigstream/rig-kafka): reconciled
2026-01-21 15:27:10 INFO  ClusterOperator:135 - Triggering periodic reconciliation for namespace rigstream
2026-01-21 15:27:10 INFO  AbstractOperator:265 - Reconciliation #3268(timer) Kafka(rigstream/rig-kafka): Kafka rig-kafka will be checked for creation or modification
2026-01-21 15:27:10 WARN  StatusUtils:106 - Reconciliation #3269(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the externalConfiguration property at path spec.externalConfiguration has been deprecated. The external configuration is deprecated and will be removed in the future. Please use the template section instead to configure additional environment variables or volumes.
2026-01-21 15:27:10 WARN  StatusUtils:131 - Reconciliation #3269(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the object externalConfiguration at path spec.externalConfiguration has been deprecated. This object has been replaced with KafkaConnectTemplate.
2026-01-21 15:27:10 WARN  StatusUtils:106 - Reconciliation #3269(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the env property at path spec.externalConfiguration.env has been deprecated. The external configuration environment variables are deprecated and will be removed in the future. Please use the environment variables in a container template instead.
2026-01-21 15:27:10 WARN  StatusUtils:106 - Reconciliation #3269(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the externalConfiguration property at path spec.externalConfiguration has been deprecated. The external configuration is deprecated and will be removed in the future. Please use the template section instead to configure additional environment variables or volumes.
2026-01-21 15:27:10 WARN  StatusUtils:131 - Reconciliation #3269(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the object externalConfiguration at path spec.externalConfiguration has been deprecated. This object has been replaced with KafkaConnectTemplate.
2026-01-21 15:27:10 WARN  StatusUtils:106 - Reconciliation #3269(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the env property at path spec.externalConfiguration.env has been deprecated. The external configuration environment variables are deprecated and will be removed in the future. Please use the environment variables in a container template instead.
2026-01-21 15:27:10 INFO  AbstractOperator:265 - Reconciliation #3269(timer) KafkaConnect(rigstream/rig-connect): KafkaConnect rig-connect will be checked for creation or modification
2026-01-21 15:27:11 INFO  KafkaConnectAssemblyOperator:465 - Reconciliation #3269(timer) KafkaConnect(rigstream/rig-connect): creating/updating connector: rig-jdbc-sink
2026-01-21 15:27:11 INFO  KafkaConnectAssemblyOperator:465 - Reconciliation #3269(timer) KafkaConnect(rigstream/rig-connect): creating/updating connector: rig-tool-memory-jdbc-sink
2026-01-21 15:27:11 INFO  AbstractOperator:545 - Reconciliation #3269(timer) KafkaConnect(rigstream/rig-connect): reconciled
2026-01-21 15:27:12 INFO  AbstractOperator:545 - Reconciliation #3268(timer) Kafka(rigstream/rig-kafka): reconciled
2026-01-21 15:29:10 INFO  ClusterOperator:135 - Triggering periodic reconciliation for namespace rigstream
2026-01-21 15:29:10 INFO  AbstractOperator:265 - Reconciliation #3270(timer) Kafka(rigstream/rig-kafka): Kafka rig-kafka will be checked for creation or modification
2026-01-21 15:29:10 WARN  StatusUtils:106 - Reconciliation #3271(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the externalConfiguration property at path spec.externalConfiguration has been deprecated. The external configuration is deprecated and will be removed in the future. Please use the template section instead to configure additional environment variables or volumes.
2026-01-21 15:29:10 WARN  StatusUtils:131 - Reconciliation #3271(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the object externalConfiguration at path spec.externalConfiguration has been deprecated. This object has been replaced with KafkaConnectTemplate.
2026-01-21 15:29:10 WARN  StatusUtils:106 - Reconciliation #3271(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the env property at path spec.externalConfiguration.env has been deprecated. The external configuration environment variables are deprecated and will be removed in the future. Please use the environment variables in a container template instead.
2026-01-21 15:29:10 WARN  StatusUtils:106 - Reconciliation #3271(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the externalConfiguration property at path spec.externalConfiguration has been deprecated. The external configuration is deprecated and will be removed in the future. Please use the template section instead to configure additional environment variables or volumes.
2026-01-21 15:29:10 WARN  StatusUtils:131 - Reconciliation #3271(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the object externalConfiguration at path spec.externalConfiguration has been deprecated. This object has been replaced with KafkaConnectTemplate.
2026-01-21 15:29:10 WARN  StatusUtils:106 - Reconciliation #3271(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the env property at path spec.externalConfiguration.env has been deprecated. The external configuration environment variables are deprecated and will be removed in the future. Please use the environment variables in a container template instead.
2026-01-21 15:29:10 INFO  AbstractOperator:265 - Reconciliation #3271(timer) KafkaConnect(rigstream/rig-connect): KafkaConnect rig-connect will be checked for creation or modification
2026-01-21 15:29:11 INFO  KafkaConnectAssemblyOperator:465 - Reconciliation #3271(timer) KafkaConnect(rigstream/rig-connect): creating/updating connector: rig-jdbc-sink
2026-01-21 15:29:11 INFO  KafkaConnectAssemblyOperator:465 - Reconciliation #3271(timer) KafkaConnect(rigstream/rig-connect): creating/updating connector: rig-tool-memory-jdbc-sink
2026-01-21 15:29:11 INFO  AbstractOperator:545 - Reconciliation #3271(timer) KafkaConnect(rigstream/rig-connect): reconciled
2026-01-21 15:29:12 INFO  AbstractOperator:545 - Reconciliation #3270(timer) Kafka(rigstream/rig-kafka): reconciled
2026-01-21 15:30:04 INFO  StrimziPodSetController:348 - Reconciliation #3272(watch) StrimziPodSet(rigstream/rig-connect-connect): StrimziPodSet will be reconciled
2026-01-21 15:30:04 INFO  StrimziPodSetController:384 - Reconciliation #3272(watch) StrimziPodSet(rigstream/rig-connect-connect): reconciled
2026-01-21 15:30:04 INFO  StrimziPodSetController:348 - Reconciliation #3273(watch) StrimziPodSet(rigstream/rig-kafka-controller): StrimziPodSet will be reconciled
2026-01-21 15:30:04 INFO  StrimziPodSetController:384 - Reconciliation #3273(watch) StrimziPodSet(rigstream/rig-kafka-controller): reconciled
2026-01-21 15:30:04 INFO  StrimziPodSetController:348 - Reconciliation #3274(watch) StrimziPodSet(rigstream/rig-kafka-broker): StrimziPodSet will be reconciled
2026-01-21 15:30:04 INFO  StrimziPodSetController:384 - Reconciliation #3274(watch) StrimziPodSet(rigstream/rig-kafka-broker): reconciled
2026-01-21 15:30:04 INFO  StrimziPodSetController:348 - Reconciliation #3275(watch) StrimziPodSet(rigstream/rig-kafka-broker): StrimziPodSet will be reconciled
2026-01-21 15:30:04 INFO  StrimziPodSetController:384 - Reconciliation #3275(watch) StrimziPodSet(rigstream/rig-kafka-broker): reconciled
2026-01-21 15:30:04 INFO  StrimziPodSetController:348 - Reconciliation #3276(watch) StrimziPodSet(rigstream/rig-kafka-controller): StrimziPodSet will be reconciled
2026-01-21 15:30:04 INFO  StrimziPodSetController:384 - Reconciliation #3276(watch) StrimziPodSet(rigstream/rig-kafka-controller): reconciled
2026-01-21 15:30:04 INFO  StrimziPodSetController:348 - Reconciliation #3277(watch) StrimziPodSet(rigstream/rig-kafka-broker): StrimziPodSet will be reconciled
2026-01-21 15:30:04 INFO  StrimziPodSetController:384 - Reconciliation #3277(watch) StrimziPodSet(rigstream/rig-kafka-broker): reconciled
2026-01-21 15:30:04 INFO  StrimziPodSetController:348 - Reconciliation #3278(watch) StrimziPodSet(rigstream/rig-connect-connect): StrimziPodSet will be reconciled
2026-01-21 15:30:04 INFO  StrimziPodSetController:384 - Reconciliation #3278(watch) StrimziPodSet(rigstream/rig-connect-connect): reconciled
2026-01-21 15:31:10 INFO  ClusterOperator:135 - Triggering periodic reconciliation for namespace rigstream
2026-01-21 15:31:10 INFO  AbstractOperator:265 - Reconciliation #3279(timer) Kafka(rigstream/rig-kafka): Kafka rig-kafka will be checked for creation or modification
2026-01-21 15:31:10 WARN  StatusUtils:106 - Reconciliation #3280(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the externalConfiguration property at path spec.externalConfiguration has been deprecated. The external configuration is deprecated and will be removed in the future. Please use the template section instead to configure additional environment variables or volumes.
2026-01-21 15:31:10 WARN  StatusUtils:131 - Reconciliation #3280(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the object externalConfiguration at path spec.externalConfiguration has been deprecated. This object has been replaced with KafkaConnectTemplate.
2026-01-21 15:31:10 WARN  StatusUtils:106 - Reconciliation #3280(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the env property at path spec.externalConfiguration.env has been deprecated. The external configuration environment variables are deprecated and will be removed in the future. Please use the environment variables in a container template instead.
2026-01-21 15:31:10 WARN  StatusUtils:106 - Reconciliation #3280(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the externalConfiguration property at path spec.externalConfiguration has been deprecated. The external configuration is deprecated and will be removed in the future. Please use the template section instead to configure additional environment variables or volumes.
2026-01-21 15:31:10 WARN  StatusUtils:131 - Reconciliation #3280(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the object externalConfiguration at path spec.externalConfiguration has been deprecated. This object has been replaced with KafkaConnectTemplate.
2026-01-21 15:31:10 WARN  StatusUtils:106 - Reconciliation #3280(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the env property at path spec.externalConfiguration.env has been deprecated. The external configuration environment variables are deprecated and will be removed in the future. Please use the environment variables in a container template instead.
2026-01-21 15:31:10 INFO  AbstractOperator:265 - Reconciliation #3280(timer) KafkaConnect(rigstream/rig-connect): KafkaConnect rig-connect will be checked for creation or modification
2026-01-21 15:31:11 INFO  KafkaConnectAssemblyOperator:465 - Reconciliation #3280(timer) KafkaConnect(rigstream/rig-connect): creating/updating connector: rig-jdbc-sink
2026-01-21 15:31:11 INFO  KafkaConnectAssemblyOperator:465 - Reconciliation #3280(timer) KafkaConnect(rigstream/rig-connect): creating/updating connector: rig-tool-memory-jdbc-sink
2026-01-21 15:31:11 INFO  AbstractOperator:545 - Reconciliation #3280(timer) KafkaConnect(rigstream/rig-connect): reconciled
2026-01-21 15:31:12 INFO  AbstractOperator:545 - Reconciliation #3279(timer) Kafka(rigstream/rig-kafka): reconciled
2026-01-21 15:33:10 INFO  ClusterOperator:135 - Triggering periodic reconciliation for namespace rigstream
2026-01-21 15:33:10 INFO  AbstractOperator:265 - Reconciliation #3281(timer) Kafka(rigstream/rig-kafka): Kafka rig-kafka will be checked for creation or modification
2026-01-21 15:33:10 WARN  StatusUtils:106 - Reconciliation #3282(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the externalConfiguration property at path spec.externalConfiguration has been deprecated. The external configuration is deprecated and will be removed in the future. Please use the template section instead to configure additional environment variables or volumes.
2026-01-21 15:33:10 WARN  StatusUtils:131 - Reconciliation #3282(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the object externalConfiguration at path spec.externalConfiguration has been deprecated. This object has been replaced with KafkaConnectTemplate.
2026-01-21 15:33:10 WARN  StatusUtils:106 - Reconciliation #3282(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the env property at path spec.externalConfiguration.env has been deprecated. The external configuration environment variables are deprecated and will be removed in the future. Please use the environment variables in a container template instead.
2026-01-21 15:33:10 WARN  StatusUtils:106 - Reconciliation #3282(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the externalConfiguration property at path spec.externalConfiguration has been deprecated. The external configuration is deprecated and will be removed in the future. Please use the template section instead to configure additional environment variables or volumes.
2026-01-21 15:33:10 WARN  StatusUtils:131 - Reconciliation #3282(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the object externalConfiguration at path spec.externalConfiguration has been deprecated. This object has been replaced with KafkaConnectTemplate.
2026-01-21 15:33:10 WARN  StatusUtils:106 - Reconciliation #3282(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the env property at path spec.externalConfiguration.env has been deprecated. The external configuration environment variables are deprecated and will be removed in the future. Please use the environment variables in a container template instead.
2026-01-21 15:33:10 INFO  AbstractOperator:265 - Reconciliation #3282(timer) KafkaConnect(rigstream/rig-connect): KafkaConnect rig-connect will be checked for creation or modification
2026-01-21 15:33:11 INFO  KafkaConnectAssemblyOperator:465 - Reconciliation #3282(timer) KafkaConnect(rigstream/rig-connect): creating/updating connector: rig-jdbc-sink
2026-01-21 15:33:11 INFO  KafkaConnectAssemblyOperator:465 - Reconciliation #3282(timer) KafkaConnect(rigstream/rig-connect): creating/updating connector: rig-tool-memory-jdbc-sink
2026-01-21 15:33:11 INFO  AbstractOperator:545 - Reconciliation #3282(timer) KafkaConnect(rigstream/rig-connect): reconciled
2026-01-21 15:33:12 INFO  AbstractOperator:545 - Reconciliation #3281(timer) Kafka(rigstream/rig-kafka): reconciled
2026-01-21 15:35:10 INFO  ClusterOperator:135 - Triggering periodic reconciliation for namespace rigstream
2026-01-21 15:35:10 INFO  AbstractOperator:265 - Reconciliation #3283(timer) Kafka(rigstream/rig-kafka): Kafka rig-kafka will be checked for creation or modification
2026-01-21 15:35:10 WARN  StatusUtils:106 - Reconciliation #3284(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the externalConfiguration property at path spec.externalConfiguration has been deprecated. The external configuration is deprecated and will be removed in the future. Please use the template section instead to configure additional environment variables or volumes.
2026-01-21 15:35:10 WARN  StatusUtils:131 - Reconciliation #3284(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the object externalConfiguration at path spec.externalConfiguration has been deprecated. This object has been replaced with KafkaConnectTemplate.
2026-01-21 15:35:10 WARN  StatusUtils:106 - Reconciliation #3284(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the env property at path spec.externalConfiguration.env has been deprecated. The external configuration environment variables are deprecated and will be removed in the future. Please use the environment variables in a container template instead.
2026-01-21 15:35:10 WARN  StatusUtils:106 - Reconciliation #3284(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the externalConfiguration property at path spec.externalConfiguration has been deprecated. The external configuration is deprecated and will be removed in the future. Please use the template section instead to configure additional environment variables or volumes.
2026-01-21 15:35:10 WARN  StatusUtils:131 - Reconciliation #3284(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the object externalConfiguration at path spec.externalConfiguration has been deprecated. This object has been replaced with KafkaConnectTemplate.
2026-01-21 15:35:10 WARN  StatusUtils:106 - Reconciliation #3284(timer) KafkaConnect(rigstream/rig-connect): In resource KafkaConnect(rigstream/rig-connect) in API version kafka.strimzi.io/v1beta2 the env property at path spec.externalConfiguration.env has been deprecated. The external configuration environment variables are deprecated and will be removed in the future. Please use the environment variables in a container template instead.
2026-01-21 15:35:10 INFO  AbstractOperator:265 - Reconciliation #3284(timer) KafkaConnect(rigstream/rig-connect): KafkaConnect rig-connect will be checked for creation or modification
2026-01-21 15:35:11 INFO  KafkaConnectAssemblyOperator:465 - Reconciliation #3284(timer) KafkaConnect(rigstream/rig-connect): creating/updating connector: rig-jdbc-sink
2026-01-21 15:35:11 INFO  KafkaConnectAssemblyOperator:465 - Reconciliation #3284(timer) KafkaConnect(rigstream/rig-connect): creating/updating connector: rig-tool-memory-jdbc-sink
2026-01-21 15:35:11 INFO  AbstractOperator:545 - Reconciliation #3284(timer) KafkaConnect(rigstream/rig-connect): reconciled
2026-01-21 15:35:12 INFO  AbstractOperator:545 - Reconciliation #3283(timer) Kafka(rigstream/rig-kafka): reconciled

Kafka-connect-metrics-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-connect-metrics
  namespace: rigstream
  labels:
    app: strimzi
data:
  kafka-connect-metrics-config.yaml: |
    lowercaseOutputName: true
    lowercaseOutputLabelNames: true
    rules:
      #kafka.connect:type=app-info,client-id="{clientid}"
      #kafka.consumer:type=app-info,client-id="{clientid}"
      #kafka.producer:type=app-info,client-id="{clientid}"
      - pattern: "kafka.(.+)<type=app-info, client-id=(.+)><>start-time-ms"
        name: kafka_$1_start_time_seconds
        labels:
          clientId: "$2"
        help: "Kafka $1 JMX metric start time seconds"
        type: GAUGE
        valueFactor: 0.001
      - pattern: "kafka.(.+)<type=app-info, client-id=(.+)><>(commit-id|version): (.+)"
        name: kafka_$1_$3_info
        value: 1
        labels:
          clientId: "$2"
          $3: "$4"
        help: "Kafka $1 JMX metric info version and commit-id"
        type: GAUGE

      #kafka.producer:type=producer-topic-metrics,client-id="{clientid}",topic="{topic}"", partition="{partition}"
      #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}",topic="{topic}"", partition="{partition}"
      - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), topic=(.+), partition=(.+)><>(.+-total|compression-rate|.+-avg|.+-replica|.+-lag|.+-lead)
        name: kafka_$2_$6
        labels:
          clientId: "$3"
          topic: "$4"
          partition: "$5"
        help: "Kafka $1 JMX metric type $2"
        type: GAUGE

      #kafka.producer:type=producer-topic-metrics,client-id="{clientid}",topic="{topic}"
      #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}",topic="{topic}"", partition="{partition}"
      - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), topic=(.+)><>(.+-total|compression-rate|.+-avg)
        name: kafka_$2_$5
        labels:
          clientId: "$3"
          topic: "$4"
        help: "Kafka $1 JMX metric type $2"
        type: GAUGE

      #kafka.connect:type=connect-node-metrics,client-id="{clientid}",node-id="{nodeid}"
      #kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id="{nodeid}"
      - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.+), node-id=(.+)><>(.+-total|.+-avg)
        name: kafka_$2_$5
        labels:
          clientId: "$3"
          nodeId: "$4"
        help: "Kafka $1 JMX metric type $2"
        type: UNTYPED

      #kafka.connect:type=kafka-metrics-count,client-id="{clientid}"
      #kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{clientid}"
      #kafka.consumer:type=consumer-coordinator-metrics,client-id="{clientid}"
      #kafka.consumer:type=consumer-metrics,client-id="{clientid}"
      - pattern: kafka.(.+)<type=(.+)-metrics, client-id=(.*)><>(.+-total|.+-avg|.+-bytes|.+-count|.+-ratio|.+-age|.+-flight|.+-threads|.+-connectors|.+-tasks|.+-ago)
        name: kafka_$2_$4
        labels:
          clientId: "$3"
        help: "Kafka $1 JMX metric type $2"
        type: GAUGE

      #kafka.connect:type=connector-task-metrics,connector="{connector}",task="{task}<> status"
      - pattern: "kafka.connect<type=connector-task-metrics, connector=(.+), task=(.+)><>status: ([a-z-]+)"
        name: kafka_connect_connector_status
        value: 1
        labels:
          connector: "$1"
          task: "$2"
          status: "$3"
        help: "Kafka Connect JMX Connector status"
        type: GAUGE

      #kafka.connect:type=task-error-metrics,connector="{connector}",task="{task}"
      #kafka.connect:type=source-task-metrics,connector="{connector}",task="{task}"
      #kafka.connect:type=sink-task-metrics,connector="{connector}",task="{task}"
      #kafka.connect:type=connector-task-metrics,connector="{connector}",task="{task}"
      - pattern: kafka.connect<type=(.+)-metrics, connector=(.+), task=(.+)><>(.+-total|.+-count|.+-ms|.+-ratio|.+-avg|.+-failures|.+-requests|.+-timestamp|.+-logged|.+-errors|.+-retries|.+-skipped)
        name: kafka_connect_$1_$4
        labels:
          connector: "$2"
          task: "$3"
        help: "Kafka Connect JMX metric type $1"
        type: GAUGE

      #kafka.connect:type=connector-metrics,connector="{connector}"
      #kafka.connect:type=connect-worker-metrics,connector="{connector}"
      - pattern: kafka.connect<type=connect-worker-metrics, connector=(.+)><>([a-z-]+)
        name: kafka_connect_worker_$2
        labels:
          connector: "$1"
        help: "Kafka Connect JMX metric $1"
        type: GAUGE

      #kafka.connect:type=connect-worker-metrics
      - pattern: kafka.connect<type=connect-worker-metrics><>([a-z-]+)
        name: kafka_connect_worker_$1
        help: "Kafka Connect JMX metric worker"
        type: GAUGE

      #kafka.connect:type=connect-worker-rebalance-metrics
      - pattern: kafka.connect<type=connect-worker-rebalance-metrics><>([a-z-]+)
        name: kafka_connect_worker_rebalance_$1
        help: "Kafka Connect JMX metric rebalance information"
        type: GAUGE

Kafka Connect Custom Image Build

Image Base

• Base Image: quay.io/strimzi/kafka:0.47.0-kafka-4.0.0
• Kafka Version: 4.0.0
• Strimzi Version: 0.47.0

Build Steps

1. Prepare build directory with Dockerfile and plugin JARs
2. Build image:

   docker build -t harbor.sperrydigital.local/righive/kafka-connect-rig-sink:latest .

3. Push to registry:

   docker push harbor.sperrydigital.local/righive/kafka-connect-rig-sink:latest

What the Dockerfile Does

Stage 1 - Download connectors using Confluent Hub:

• JDBC Connector 10.8.4
• Avro Converter 7.5.0

Stage 2 - Copy to Strimzi base image:

• Copy Confluent connectors to /opt/kafka/plugins/
• Add custom plugin JARs (jcustenborder, Debezium, router connector)
• Download Debezium dependencies and Groovy engine
• Add PostgreSQL JDBC driver

No modifications to entrypoint, startup scripts, or JVM arguments.

No modifications to entrypoint, startup scripts, or JVM arguments.

[scholzj]

Hmm, that looks all good. So I guess the Connect log itself is the last place which might show something useful :-(.

[ChinthapalliNikhithaChandana]

Connect pod logs.txt

[scholzj]

Wait. You said that you are using Strimzi 0.48.0 and that is what the logs suggest as well. But you build the image from Strimzi 0.47.0. That is wrong. You need to build it from 0.48.0. (it needs to be the exact same version as the operator uses)

[ChinthapalliNikhithaChandana]

I think that solved the issue. Thanks a lot.