docker-compose 搭建 kafka集群

搭建集群

docker-compose up -d --scale kafka=2        本機啟動一個有兩個節點的 Kafka 集群

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"

  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092"             # kafka 把9092端口隨機映射到主機的端口
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 192.168.199.163           #本機ip
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: test:1:1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

輸出結果

 chen@macos  ~/docker/kafka01 sudo docker ps                                                                                                                                                   
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                                NAMES
4ede7dc26df7        wurstmeister/zookeeper   "/bin/sh -c '/usr/sb…"   22 minutes ago      Up 22 minutes       22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   kafka01_zookeeper_1
2f7c0a71af03        wurstmeister/kafka       "start-kafka.sh"         22 minutes ago      Up 22 minutes       0.0.0.0:32772->9092/tcp                              kafka01_kafka_2
e0b72d9466e0        wurstmeister/kafka       "start-kafka.sh"         22 minutes ago      Up 22 minutes       0.0.0.0:32771->9092/tcp                              kafka01_kafka_1 

注意：docker-compose up -d --scale kafka=2        # 可以進行擴展或縮容                                                                                                    

基礎操作

列出主題

容器內部查詢主題
bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --describe --zookeeper kafka01_zookeeper_1:2181          #可以通過容器的名字或者IP訪問
Topic:test	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: test	Partition: 0	Leader: 1002	Replicas: 1002	Isr: 1002
容器外部或內部部查詢主題
bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh  --describe --zookeeper 192.168.199.163:2181          #這里因為zookeeper映射到了主題的2181端口
Topic:test	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: test	Partition: 0	Leader: 1002	Replicas: 1002	Isr: 1002

創建主題

bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh  --create --topic chat --partitions 3 --zookeeper 192.168.199.163:2181 --replication-factor 2
Created topic chat.
bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --describe --zookeeper kafka01_zookeeper_1:2181
Topic:chat	PartitionCount:3	ReplicationFactor:2	Configs:
	Topic: chat	Partition: 0	Leader: 1002	Replicas: 1002,1001	Isr: 1002,1001
	Topic: chat	Partition: 1	Leader: 1001	Replicas: 1001,1002	Isr: 1001,1002
	Topic: chat	Partition: 2	Leader: 1002	Replicas: 1002,1001	Isr: 1002,1001
Topic:test	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: test	Partition: 0	Leader: 1002	Replicas: 1002	Isr: 1002

消息消費

生產者－容器里面
bash-4.4# ./kafka-console-producer.sh --broker-list localhost:9092 --topic chat
>da
>

消費者－容器里面
/opt/kafka_2.12-2.2.0/bin # ./kafka-console-consumer.sh  --bootstrap-server localhost:9092 --topic chat --from-beginning
da

注意：上面要在宿主機上只要將localhost:9092換為kafka暴露在主機上端口即可