Custom Dockerfile and docker-compose.yml file for Kafka and Zookeeper containers doesn't produce and consume messages

Question

I'm trying to create a custom Dockerfile and docker-compose.yml files for Kafka and Zookeeper services, in order to understand how to configure it, and how those services are communicated between each other.

But the connection doesn't seems to be working when I'm connecting to the Broker URL using a Python script. Specifically, the execution of the producer and the consumer is not sending the messages to the respective topic, and because of that, the consumer is not handling it.

First, I'm going to share the Dockerfile:

FROM ubuntu:latest

ARG DEBIAN_FRONTEND=noninteractive

WORKDIR /home/ubuntu/

RUN apt-get update && \
    apt-get install openjdk-8-jdk -y

ENV JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

RUN apt-get update && \
    apt-get install curl -y

RUN curl "https://dlcdn.apache.org/kafka/3.7.1/kafka_2.13-3.7.1.tgz" \
    --output "kafka.tgz"

RUN mkdir -p kafka && \
    tar -xzf kafka.tgz -C kafka --strip 1 && \
    rm kafka.tgz

ENV PATH="/home/ubuntu/kafka/bin:$PATH"

RUN mkdir -p kafka/data && \
    mkdir -p kafka/data/zookeeper && \
    mkdir -p kafka/data/server

RUN sed -i "s/dataDir=\/tmp\/zookeeper/dataDir=\/home\/ubuntu\/kafka\/data\/zookeeper/" \
    /home/ubuntu/kafka/config/zookeeper.properties

RUN sed -i "s/log.dirs=\/tmp\/kafka-logs/log.dirs=\/home\/ubuntu\/kafka\/data\/server/" \
    /home/ubuntu/kafka/config/server.properties

EXPOSE 2181 9092

ENTRYPOINT [ "" ]

Then, there is the docker-compose.yml file:

version: "3.9"

networks:
  kafka-tutorial-network:
    name: kafka-tutorial-network
    driver: bridge

volumes:
  kafka-tutorial-volume:
    name: kafka-tutorial-volume

services:
  zookeeper-service:
    container_name: zookeeper-service

    build:
      context: .
      dockerfile: Dockerfile
    
    image: kafka-tutorial-image:1.0

    command: ["zookeeper-server-start.sh", "kafka/config/zookeeper.properties"]

    networks:
      - kafka-tutorial-network
    
    ports:
      - 2181:2181
  
  kafka-service:
    container_name: kafka-service

    build:
      context: .
      dockerfile: Dockerfile
    
    image: kafka-tutorial-image:1.0

    command: > 
      bash -c "sed -i 's/zookeeper.connect=localhost:2181/zookeeper.connect=zookeeper-service:2181/' /home/ubuntu/kafka/config/server.properties
      && kafka-server-start.sh kafka/config/server.properties"
    
    volumes:
      - ./kafka_data:/home/ubuntu/kafka/data

    networks:
      - kafka-tutorial-network
    
    ports:
      - 9092:9092
    
    depends_on:
      - zookeeper-service

So, when running the following command:

> docker-compose up -d --build

Both services are up and running, with their respective logs.

enter image description here

After all this previous configuration, I created this Python script that connects to the specified Kafka Broker and produce a bunch of message, and then sent them to a topic called test-topic.

import json
from kafka import KafkaProducer
import time

producer = KafkaProducer(
    bootstrap_servers=["127.0.0.1:9092"],
    value_serializer=lambda value: json.dumps(value).encode("UTF-8")
)


if __name__ == "__main__":
    
    for number in range(1, 11):
        data_to_send = {"number": number}
        producer.send("test-topic", value=data_to_send)
        time.sleep(8)

Also I created this other Python Script that consumed the previoulsy created messages of the specified topic:

import json
from kafka import KafkaConsumer

consumer = KafkaConsumer(
    "test-topic",
    bootstrap_servers=["127.0.0.1:9092"],
    value_deserializer=lambda value: json.loads(value.decode("UTF-8"))
)

if __name__ == "__main__":
    for message in consumer:
        print(message.value)

When I run both scripts, the producer.py file seems to be working. enter image description here

But the consumer.py file seems to not have any message to be consumed. enter image description here

The breakpoint never steps into the print function: enter image description here

So, to tests if the configuration of my Docker containers is working, I create messages and consume them from the container shell, with the following commands:

> /kafka-console-producer.sh \
--broker-list localhost:9092 \
--topic getting-started --property "parse.key=true" \
--property "key.separator=:"

> foo:first message
foo:second message
bar:first message
foo:third message
bar:second message

The first command creates a topic called getting-started and the second prompt sends five messages to the respective topic.

Then to consume them, we execute the following command:

> kafka-console-consumer.sh \
> --bootstrap-server 127.0.0.1:9092 \
> --topic getting-started --group cli-consumer --from-beginning \
> --property "print.key=true" --property "key.separator=:"

And it's working: enter image description here

So, at the moment, the problem is not with the communication between the two services.

Finally, only to verify if the script is working, I connect the Python Scripts with other Docker container services generated with this docker-compose.yml file that I found in a post:

version: "3"
networks:
  myNetwork:

services:

  zookeeper:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
    networks:
      - myNetwork

  kafka:
    image: 'bitnami/kafka:latest'
    user: root
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_LISTENERS=PLAINTEXT://:9092
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    volumes:
      - ./Kafka:/bitnami/kafka
    networks:
      - myNetwork
    depends_on:
      - zookeeper        

And with this configuration, the Python Scripts seems to be working as expected: enter image description here

Can someone tellm what I'm doing wrong when creating the configuration or the networking inside the Dockerfile or the docker-compose.yml?