cp-all-in-one-cloud Connect admin calls timing out

[ianb-pomelo]

Description
Following the instructions/environment variables set to connect to a Confluence cloud instance of Kafka, my docker instance is timing out connecting to the admin client for the instance. The logs continually show errors like:

connect  | [2024-02-03 01:24:36,871] INFO [AdminClient clientId=adminclient-2] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
connect  | [2024-02-03 01:24:36,871] INFO [AdminClient clientId=adminclient-2] Cancelled in-flight API_VERSIONS request with correlation id 137 due to node -1 being disconnected (elapsed time since creation: 5032ms, elapsed time since send: 5032ms, request timeout: 3600000ms) (org.apache.kafka.clients.NetworkClient)
connect  | [2024-02-03 01:24:37,690] INFO [AdminClient clientId=adminclient-2] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager)
connect  | org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: fetchMetadata

I am just running the single Confluence Kafka connect container via docker compose with all environment variables from cp-all-in-one-cloud/docker-compose.connect.yml set to the correct values from the Confluence UI.

I can see in in the confluence UI that the connect config, offset and status topics are being connected (the topics exist and have the connection details and error messages from the docker container) but the connection I have setup does not propagate the messages to Confluence cloud.

I am using a docker image created by downloading and copying a jar following this: https://docs.confluent.io/platform/current/connect/extending.html#add-additional-software

Troubleshooting
Command: docker compose up

Compose File:

version: "3.7"
name: docker-connect
services:
  connect:
    image: spanner-connect
    container_name: connect
    ports:
      - "8083:8083"
    environment:
      CONNECT_BOOTSTRAP_SERVERS: XXX.gcp.confluent.cloud:9092
      CONNECT_GROUP_ID: "connect-unique"
      CONNECT_CONFIG_STORAGE_TOPIC: demo-connect-configs
      CONNECT_OFFSET_STORAGE_TOPIC: demo-connect-offsets
      CONNECT_STATUS_STORAGE_TOPIC: demo-connect-status
      CONNECT_REPLICATION_FACTOR: 3
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 3
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 3
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 3
      CONNECT_KEY_CONVERTER: "org.apache.kafka.connect.storage.StringConverter"
      CONNECT_VALUE_CONVERTER: "io.confluent.connect.avro.AvroConverter"
      CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE: "true"
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: https://XXX.gcp.confluent.cloud
      CONNECT_VALUE_CONVERTER_BASIC_AUTH_CREDENTIALS_SOURCE: USER_INFO
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO: XXX:XXXX
      CONNECT_REST_ADVERTISED_HOST_NAME: "connect"
      CONNECT_PLUGIN_PATH: "/usr/share/java,/usr/share/confluent-hub-components"
      CONNECT_LOG4J_ROOT_LOGLEVEL: INFO
      CONNECT_LOG4J_LOGGERS: org.reflections=ERROR
      # CLASSPATH required due to CC-2422
      CLASSPATH: /usr/share/java/monitoring-interceptors/monitoring-interceptors-6.2.0.jar
      # Connect worker
      CONNECT_SECURITY_PROTOCOL: SASL_SSL
      CONNECT_SASL_JAAS_CONFIG: org.apache.kafka.common.security.plain.PlainLoginModule required username="XXX" password="XXX";
      CONNECT_SASL_MECHANISM: PLAIN
      # Connect producer
      CONNECT_PRODUCER_SECURITY_PROTOCOL: SASL_SSL
      CONNECT_PRODUCER_SASL_JAAS_CONFIG: org.apache.kafka.common.security.plain.PlainLoginModule required username="XXX" password="XXX";
      CONNECT_PRODUCER_SASL_MECHANISM: PLAIN
      CONNECT_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor"
      CONNECT_PRODUCER_CONFLUENT_MONITORING_INTERCEPTOR_SECURITY_PROTOCOL: SASL_SSL
      CONNECT_PRODUCER_CONFLUENT_MONITORING_INTERCEPTOR_SASL_JAAS_CONFIG: org.apache.kafka.common.security.plain.PlainLoginModule required username="XXX" password="XXX";
      CONNECT_PRODUCER_CONFLUENT_MONITORING_INTERCEPTOR_SASL_MECHANISM: PLAIN
      # Connect consumer
      CONNECT_CONSUMER_SECURITY_PROTOCOL: SASL_SSL
      CONNECT_CONSUMER_SASL_JAAS_CONFIG: org.apache.kafka.common.security.plain.PlainLoginModule required username="XXX" password="XXX";
      CONNECT_CONSUMER_SASL_MECHANISM: PLAIN
      CONNECT_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"
      CONNECT_CONSUMER_CONFLUENT_MONITORING_INTERCEPTOR_SECURITY_PROTOCOL: SASL_SSL
      CONNECT_CONSUMER_CONFLUENT_MONITORING_INTERCEPTOR_SASL_JAAS_CONFIG: org.apache.kafka.common.security.plain.PlainLoginModule required username="XXX" password="XXX";
      CONNECT_CONSUMER_CONFLUENT_MONITORING_INTERCEPTOR_SASL_MECHANISM: PLAIN
    volumes:
        - secretFile:secret_file.json

Environment

• Operating System: Darwin 23.3.0 arm64
• Version of Docker: 24.0.7