Broker transport failure: localhost:9092/bootstrap

[sarthaksnh5]

2024-06-20 15:38:24 time="2024-06-20T10:08:24Z" level=error msg="received an error with code Local: Broker transport failure: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)"

2024-06-20 15:38:25 %3|1718878105.022|FAIL|rdkafka#producer-1| [thrd:0.0.0.0:9092/bootstrap]: 0.0.0.0:9092/bootstrap: Connect to ipv4#0.0.0.0:9092 failed: Connection refused (after 0ms in state CONNECT, 11 identical error(s) suppressed)

[fkarakas]

?

[sarthaksnh5]

I am using apache-kafka executed through the folowing command:
docker run -p 9092:9092 apache/kafka:3.7.0

When I try to run the kafka-scheduler
docker run -e BOOTSTRAP_SERVERS="localhost:9092" etf1/kafka-message-scheduler

I am getting the above error

[fkarakas]

when you are running the scheduler in a container and defining "localhost:9092" in bootstrap_servers variable, it will try to resolve "localhost" inside the container and not on your computer.

i recommend you to use a docker compose to run the stack locally

services:

  scheduler:
    image: etf1/kafka-message-scheduler:v0.0.6
    depends_on:
      init-kafka:
        condition: service_completed_successfully
    ports:
      - 8000:8000
    environment:
      BOOTSTRAP_SERVERS: kafka:29092
    healthcheck:
      test: wget --no-verbose --tries=1 --spider http://localhost:8000/schedules || exit 1
      interval: 60s
      retries: 5
      start_period: 20s
      timeout: 10s
      
  zookeeper:
    image: confluentinc/cp-zookeeper
    ports:
      - 2181:2181
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      # for health check "ruok"
      KAFKA_OPTS: "-Dzookeeper.4lw.commands.whitelist=*"
    healthcheck:
      test: ["CMD-SHELL", "echo 'ruok' | nc -w 2 localhost 2181 | grep imok"]
      interval: 20s
      timeout: 10s
      retries: 10

  kafka:
    image: confluentinc/cp-kafka
    depends_on:
      zookeeper:
        condition: service_healthy
    healthcheck:
      test: ["CMD-SHELL", "kafka-topics --bootstrap-server localhost:9092 --list"]
      interval: 30s
      timeout: 10s
      retries: 20
    ports:
      # Exposes 9092 for external connections to the broker
      # Use kafka:29092 for connections internal on the docker network
      # See https://rmoff.net/2018/08/02/kafka-listeners-explained/ for details
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'

  init-kafka:
    image: confluentinc/cp-kafka
    depends_on:
      kafka:
        condition: service_healthy
    entrypoint: [ '/bin/sh', '-c' ]
    command: |
      "
      # blocks until kafka is reachable
      kafka-topics --bootstrap-server kafka:29092 --list

      echo -e 'Creating kafka topics'
      kafka-topics --bootstrap-server kafka:29092 --create --if-not-exists --topic schedules --config "cleanup.policy=compact" --config "retention.ms=-1"
      kafka-topics --bootstrap-server kafka:29092 --create --if-not-exists --topic history

      echo -e 'Successfully created the following topics:'
      kafka-topics --bootstrap-server kafka:29092 --list
      "

start command

docker-compose -f docker-compose.local.yml up -d

stop command

docker-compose -f docker-compose.local.yml down -v

Hopes this will help you, i will add in the documentation the docker compose

[sarthaksnh5]

Hello @fkarakas

It is working perfectly.

[sarthaksnh5]

Hello @fkarakas

I am facing the below error in kafka scheduler admin

failed to start runner: cannot create bleve db ./.db/: cannot create new index, path already exists
etf1/kafka-message-scheduler-admin#5

I have created the issue for the same in the repository, can you please help