基於Docker的Kafka部署

一 准備

1.1 安裝docker-dompose

#部署
sudo curl -L "https://github.com/docker/compose/releases/download/1.24.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
#設置權限
chmod +x /usr/bin/docker-compose
#驗證
docker-compose version

　　

1.2 下載鏡像

docker pull wurstmeister/zookeeper:latest
docker pull wurstmeister/kafka:0.10.0.1

　　

也可以使用Dockerfile構建，構建文件都在Github上https://github.com/jdlzy/kafka-docker

二 部署Zookeeper

在hub.docker.com網站上，Star最多的kafka鏡像是wurstmeister/kafka

https://github.com/wurstmeister/kafka-docker

首先需要新建專用網絡

docker network create --driver bridge --subnet 172.23.0.0/25 --gateway 172.23.0.1 zookeeper_network

部署zookeeper需要在任意目錄下新建docker-compose.yml，將下邊的內容拷貝到目錄中。

#前台啟動
docker-compose up
#后台啟動
docker-compose up -d

2.1 yml文件

version: '3.4'
services:
  zoo1:
    image: wurstmeister/zookeeper
    restart: always
    hostname: zookeeper
    container_name: zookeeper
    ports:
    - 2181:2181
    volumes:
    - "/root/lzy/kafka-docker/data/kafka/data:/data"
    - "/root/lzy/kafka-docker/data/kafka/datalog:/datalog"
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=zookeeper:2888:3888
    networks:
      default:
        ipv4_address: 172.23.0.11
networks:
  default:
    external:
      name: zookeeper_network

三 Kafka部署

在任意目錄下新建docker-compose.yml，將下邊的內容拷貝到目錄中。

#前台啟動
docker-compose up
#后台啟動
docker-compose up -d

3.1 yml文件

10.95.3.172是我的物理機的IP，這里穿件3個節點，Broker端口分別為9092、9093、9094

version: '2'
services:
  kafka1:
    image: wurstmeister/kafka:0.10.0.1
    restart: always
    hostname: kafka1
    container_name: kafka1
    ports:
    - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://10.95.3.172:9092
      KAFKA_ADVERTISED_HOST_NAME: 10.95.3.172
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    external_links:
    - zookeeper
    networks:
      default:
        ipv4_address: 172.23.0.14
  kafka2:
    image: wurstmeister/kafka:0.10.0.1
    restart: always
    hostname: kafka2
    container_name: kafka2
    ports:
    - "9093:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://10.95.3.172:9093 
      KAFKA_ADVERTISED_HOST_NAME: 10.95.3.172
      KAFKA_ADVERTISED_PORT: 9093
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    external_links:  # 連接本compose文件以外的container
    - zookeeper
    networks:
      default:
        ipv4_address: 172.23.0.15
  kafka3:
    image: wurstmeister/kafka:0.10.0.1
    restart: always
    hostname: kafka3
    container_name: kafka3
    ports:
    - "9094:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://10.95.3.172:9094
      KAFKA_ADVERTISED_HOST_NAME: 10.95.3.172
      KAFKA_ADVERTISED_PORT: 9094
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    external_links:  # 連接本compose文件以外的container
    - zookeeper
    networks:
      default:
        ipv4_address: 172.23.0.16
networks:
  default:
    external:   # 使用已創建的網絡
      name: zookeeper_network

 

四 測試

在任意物理機（需要部署了wurstmeister/kafka:0.10.0.1鏡像），或者直接下載0.10.0.1的Kafka安裝包。 
如果是基於Docker，執行下邊的命令啟動一個臨時容器

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -e HOST_IP=$1 -e ZK=$2 -i -t wurstmeister/kafka:0.10.0.1 /bin/bash

如果是基於Kafka安裝包，直接解壓即可。 

然后到/opt/kafka/bin目錄下

4.1 創建Topic

–replication-factor表示副本數，–partitions表示分片數量

./kafka-topics.sh --create --zookeeper 10.95.3.172:2181 --replication-factor 2 --partitions 3 --topic t1
#查看topic
./kafka-topics.sh --list --zookeeper 10.95.3.172:2181

 

4.2 創建生產者

./kafka-console-producer.sh --broker-list 10.95.3.172:9092,10.95.3.172:9093,10.95.3.172:9094 --topic t1

 

4.3 創建消費者

–from-beginning表示從消息起始開始讀取

./kafka-console-consumer.sh --zookeeper 10.95.3.172:2181 --topic t1 --from-beginning

 

五 非docker-compose部署方式

不使用docker-compose可以進行分布式部署，但是目前還有問題，有待進一步測試

5.1 部署zookeeper

docker run -d --name zookeeper --publish 2181:2181 --volume /etc/localtime:/etc/localtime wurstmeister/zookeeper:latest

5.2 部署kafka

創建多節點時候需要修改KAFKA_BROKER_ID為唯一， 
-p映射到物理機的端口不能重復，第二個可以為9093，第三個可以為9094等。

docker run -d --name kafka \
-e HOST_IP=10.95.3.172 \
-e KAFKA_ADVERTISED_PORT=9092 \
--env KAFKA_ADVERTISED_HOST_NAME=10.95.3.172 \
-e KAFKA_BROKER_ID=1 \
-e ZK=zk \
-p 9092:9092 \
--link zookeeper:zk \
-t wurstmeister/kafka:0.10.0.1

 轉載：http://blog.leanote.com/archives/kobeliuziyang