kafka 依賴 zookeeper
[root@master001 docker_test]# docker search zookeeper NAME DESCRIPTION STARS OFFICIAL AUTOMATED zookeeper Apache ZooKeeper is an open-source server wh… 807 [OK] jplock/zookeeper Builds a docker image for Zookeeper version … 164 [OK] wurstmeister/zookeeper 108 [OK] mesoscloud/zookeeper ZooKeeper 73 [OK] bitnami/zookeeper ZooKeeper is a centralized service for distr… 27 [OK] mbabineau/zookeeper-exhibitor 24 [OK] digitalwonderland/zookeeper Latest Zookeeper - clusterable 20 [OK] tobilg/zookeeper-webui Docker image for using `zk-web` as ZooKeeper… 14 [OK] confluent/zookeeper 13 [OK] debezium/zookeeper Zookeeper image required when running the De… 10 [OK] 31z4/zookeeper Dockerized Apache Zookeeper. 6 [OK] thefactory/zookeeper-exhibitor Exhibitor-managed ZooKeeper with S3 backups … 6 [OK] engapa/zookeeper Zookeeper image optimised for being used int… 2 emccorp/zookeeper Zookeeper 2 openshift/zookeeper-346-fedora20 ZooKeeper 3.4.6 with replication support 1 strimzi/zookeeper 1 duffqiu/zookeeper-cli 1 [OK] paulbrown/zookeeper Zookeeper on Kubernetes (PetSet) 1 [OK] josdotso/zookeeper-exporter ref: https://github.com/carlpett/zookeeper_e… 1 [OK] perrykim/zookeeper k8s - zookeeper ( forked k8s contrib ) 1 [OK] dabealu/zookeeper-exporter zookeeper exporter for prometheus 0 [OK] midonet/zookeeper Dockerfile for a Zookeeper server. 0 [OK] humio/zookeeper-dev zookeeper build with zulu jvm. 0 phenompeople/zookeeper Apache ZooKeeper is an open-source server wh… 0 [OK] avvo/zookeeper Apache Zookeeper 0 [OK]
[root@master001 docker_test]# docker pull zookeeper Using default tag: latest latest: Pulling from library/zookeeper 68ced04f60ab: Pull complete 4874c5772968: Pull complete 1036c6da18fe: Pull complete 91a28e49706a: Pull complete 493d020c97b2: Pull complete 0608ac18b7bd: Pull complete e3602b3be15f: Pull complete 5b9f8d3adfc2: Pull complete Digest: sha256:a8003722a5d8886a4c10e217d194c168cee1291cd0287a2b7d7af58476e84f03 Status: Downloaded newer image for zookeeper:latest
[root@master001 docker_test]# docker search kafka NAME DESCRIPTION STARS OFFICIAL AUTOMATED wurstmeister/kafka Multi-Broker Apache Kafka Image 1096 [OK] spotify/kafka A simple docker image with both Kafka and Zo… 391 [OK] sheepkiller/kafka-manager kafka-manager 183 [OK] bitnami/kafka Apache Kafka is a distributed streaming plat… 123 [OK] ches/kafka Apache Kafka. Tagged versions. JMX. Cluster-… 115 [OK] hlebalbau/kafka-manager CMAK (previous known as Kafka Manager) As Do… 53 [OK] kafkamanager/kafka-manager Docker image for Kafka manager 44 landoop/kafka-topics-ui UI for viewing Kafka Topics config and data … 33 [OK] solsson/kafka http://kafka.apache.org/documentation.html#q… 18 [OK] landoop/kafka-lenses-dev Lenses with Kafka. +Connect +Generators +Con… 18 [OK] johnnypark/kafka-zookeeper Kafka and Zookeeper combined image 17 debezium/kafka Kafka image required when running the Debezi… 15 [OK] digitalwonderland/kafka Latest Kafka - clusterable 15 [OK] danielqsj/kafka-exporter Kafka exporter for Prometheus 14 [OK] landoop/kafka-connect-ui Web based UI for Kafka Connect. 10 [OK]
[root@master001 docker_test]# docker pull wurstmeister/kafka Using default tag: latest latest: Pulling from wurstmeister/kafka e7c96db7181b: Pull complete f910a506b6cb: Pull complete b6abafe80f63: Pull complete b7af7deddd9e: Pull complete e5ab8db747c6: Pull complete 81a231196d32: Pull complete Digest: sha256:47bb92ce7ff3d5829a0f02ddd7ab1d13ebd6c71db445576a4998d07cf1c309da Status: Downloaded newer image for wurstmeister/kafka:latest
運行容器 zookeeper:
[root@master001 docker_test]# docker run -d --restart=always --log-driver json-file --log-opt max-size=100m --log-opt max-file=2 --name zookeeper -p 2181:2181 -v /etc/localtime:/etc/localtime zookeeper 1026a8fd54784c2ed91abac184de6eec712e79e6fed448696d505ed862a0dc2d
運行 容器 kafka:
[root@master001 docker_test]# docker run -d --restart=always --log-driver json-file --log-opt max-size=100m --log-opt max-file=2 --name kafka -p 9092:9092 -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT=192.168.205.69:2181/kafka -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.205.69:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -v /etc/localtime:/etc/localtime wurstmeister/kafka
參數說明： -e KAFKA_BROKER_ID=0 在kafka集群中，每個kafka都有一個BROKER_ID來區分自己 -e KAFKA_ZOOKEEPER_CONNECT=192.168.205.69:2181/kafka 配置zookeeper管理kafka的路徑162.168.205.69:2181/kafka -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.205.69:9092 把kafka的地址端口注冊給zookeeper，如果是遠程訪問要改成外網IP,類如Java程序訪問出現無法連接。 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 配置kafka的監聽端口 -v /etc/localtime:/etc/localtime 容器時間同步虛擬機的時間
驗證kafka是否可以使用
進入容器：
[root@master001 docker_test]# docker exec -it kafka bash bash-4.4# bash-4.4# bash-4.4# bash-4.4#
bash-4.4# cd /opt/kafka_2.12-2.4.0/ bash-4.4# pwd /opt/kafka_2.12-2.4.0
bash-4.4# cd bin bash-4.4# pwd /opt/kafka_2.12-2.4.0/bin
bash-4.4# ls -l total 140 -rwxr-xr-x 1 root root 1421 Dec 9 11:46 connect-distributed.sh -rwxr-xr-x 1 root root 1394 Dec 9 11:46 connect-mirror-maker.sh -rwxr-xr-x 1 root root 1418 Dec 9 11:46 connect-standalone.sh -rwxr-xr-x 1 root root 861 Dec 9 11:46 kafka-acls.sh -rwxr-xr-x 1 root root 873 Dec 9 11:46 kafka-broker-api-versions.sh -rwxr-xr-x 1 root root 864 Dec 9 11:46 kafka-configs.sh -rwxr-xr-x 1 root root 945 Dec 9 11:46 kafka-console-consumer.sh -rwxr-xr-x 1 root root 944 Dec 9 11:46 kafka-console-producer.sh -rwxr-xr-x 1 root root 871 Dec 9 11:46 kafka-consumer-groups.sh -rwxr-xr-x 1 root root 948 Dec 9 11:46 kafka-consumer-perf-test.sh -rwxr-xr-x 1 root root 871 Dec 9 11:46 kafka-delegation-tokens.sh -rwxr-xr-x 1 root root 869 Dec 9 11:46 kafka-delete-records.sh -rwxr-xr-x 1 root root 866 Dec 9 11:46 kafka-dump-log.sh -rwxr-xr-x 1 root root 870 Dec 9 11:46 kafka-leader-election.sh -rwxr-xr-x 1 root root 863 Dec 9 11:46 kafka-log-dirs.sh -rwxr-xr-x 1 root root 862 Dec 9 11:46 kafka-mirror-maker.sh -rwxr-xr-x 1 root root 886 Dec 9 11:46 kafka-preferred-replica-election.sh -rwxr-xr-x 1 root root 959 Dec 9 11:46 kafka-producer-perf-test.sh -rwxr-xr-x 1 root root 874 Dec 9 11:46 kafka-reassign-partitions.sh -rwxr-xr-x 1 root root 874 Dec 9 11:46 kafka-replica-verification.sh -rwxr-xr-x 1 root root 9633 Dec 9 11:46 kafka-run-class.sh -rwxr-xr-x 1 root root 1376 Dec 9 11:46 kafka-server-start.sh -rwxr-xr-x 1 root root 997 Dec 9 11:46 kafka-server-stop.sh -rwxr-xr-x 1 root root 945 Dec 9 11:46 kafka-streams-application-reset.sh -rwxr-xr-x 1 root root 863 Dec 9 11:46 kafka-topics.sh -rwxr-xr-x 1 root root 958 Dec 9 11:46 kafka-verifiable-consumer.sh -rwxr-xr-x 1 root root 958 Dec 9 11:46 kafka-verifiable-producer.sh -rwxr-xr-x 1 root root 1722 Dec 9 11:46 trogdor.sh drwxr-xr-x 2 root root 4096 Dec 23 03:06 windows -rwxr-xr-x 1 root root 867 Dec 9 11:46 zookeeper-security-migration.sh -rwxr-xr-x 1 root root 1393 Dec 9 11:46 zookeeper-server-start.sh -rwxr-xr-x 1 root root 1001 Dec 9 11:46 zookeeper-server-stop.sh -rwxr-xr-x 1 root root 968 Dec 9 11:46 zookeeper-shell.sh
運行kafka生產者發送消息：
bash-4.4# ./kafka-console-producer.sh --broker-list localhost:9092 --topic shijiu520 >helloworld >
運行消費者:
bash-4.4# cd /opt/kafka_2.12-2.4.0/bin/ bash-4.4# ls connect-distributed.sh kafka-console-consumer.sh kafka-dump-log.sh kafka-reassign-partitions.sh kafka-topics.sh zookeeper-server-start.sh connect-mirror-maker.sh kafka-console-producer.sh kafka-leader-election.sh kafka-replica-verification.sh kafka-verifiable-consumer.sh zookeeper-server-stop.sh connect-standalone.sh kafka-consumer-groups.sh kafka-log-dirs.sh kafka-run-class.sh kafka-verifiable-producer.sh zookeeper-shell.sh kafka-acls.sh kafka-consumer-perf-test.sh kafka-mirror-maker.sh kafka-server-start.sh trogdor.sh kafka-broker-api-versions.sh kafka-delegation-tokens.sh kafka-preferred-replica-election.sh kafka-server-stop.sh windows kafka-configs.sh kafka-delete-records.sh kafka-producer-perf-test.sh kafka-streams-application-reset.sh zookeeper-security-migration.sh bash-4.4# ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic shijiu520 --from-beginning [2020-03-10 06:19:41,856] WARN [Consumer clientId=consumer-console-consumer-68294-1, groupId=console-consumer-68294] Error while fetching metadata with correlation id 2 : {shijiu520=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2020-03-10 06:19:41,987] WARN [Consumer clientId=consumer-console-consumer-68294-1, groupId=console-consumer-68294] Error while fetching metadata with correlation id 4 : {shijiu520=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) helloworld
大功告成!