搭建zookeeper+kafka集群

一、環境及准備

集群環境：

 

軟件版本：

部署前操作：

關閉防火牆，關閉selinux(生產環境按需關閉或打開) 同步服務器時間，選擇公網ntpd服務器或者自建ntpd服務器 [root@es1 ~]# crontab -l #為了方便直接使用公網服務器 #update time */5 * * * * /usr/bin/rdate -s time-b.nist.gov &>/dev/null

二、zookeeper集群安裝配置

1.安裝jvm依賴環境(三台)

安裝JDK

[root@node01 ~]# rpm -ivh jdk1.8.0_162-x64.rpm #為了以后升級麻煩直接安裝1.8 Preparing... ########################################### [100%] 1:jdk1.8.0_162 ########################################### [100%]

設置Java環境

[root@node01 ~]# cat /etc/profile.d/java.sh #編輯Java環境配置文件 export JAVA_HOME=/usr/java/latest export CLASSPATH=$JAVA_HOME/lib/tools.jar export PATH=$JAVA_HOME/bin:$PATH [root@node01 ~]# . /etc/profile.d/java.sh [root@node01 ~]# java -version #確認配置 java version "1.8.0_162" Java(TM) SE Runtime Environment (build 1.8.0_162-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)

2.安裝配置zookeeper

[root@node01 ~]#wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.13/zookeeper-3.4.13.tar.gz [root@node01 ~]#tar xf zookeeper-3.4.13.tar.gz -C /usr/local [root@node01 ~]#cd /usr/local [root@node01 local]#ln -sv zookeeper-3.4.13 zookeeper [root@node01 local]#cd zookeeper/conf [root@node01 conf]# cp zoo_sample.cfg zoo.cfg [root@node01 conf]# vim zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/Data/zookeeper clientPort=2181 server.1=172.16.150.154:2888:3888 server.2=172.16.150.155:2888:3888 server.3=172.16.150.156:2888:3888
 ＃配置參數說明： tickTime:客戶端與服務器或者服務器與服務器之間每個tickTime時間就會發送一次心跳。通過心跳不僅能夠用來監聽機器的工作狀態，還可以通過心跳來控制Flower跟Leader的通信時間，默認2秒 initLimit：集群中的follower服務器(F)與leader服務器(L)之間初始連接時能容忍的最多心跳數（tickTime的數量）。 syncLimit：集群中flower服務器（F）跟leader（L）服務器之間的請求和答應最多能容忍的心跳數。 dataDir：該屬性對應的目錄是用來存放myid信息跟一些版本，日志，跟服務器唯一的ID信息等。 clientPort：客戶端連接的接口，客戶端連接zookeeper服務器的端口，zookeeper會監聽這個端口，接收客戶端的請求訪問！這個端口默認是2181。 service.N=YYY:A:B N：代表服務器編號（也就是myid里面的值） YYY：服務器地址 A：表示 Flower 跟 Leader的通信端口，簡稱服務端內部通信的端口（默認2888） B：表示 是選舉端口（默認是3888）

創建zookeeper所需要的目錄和myid文件

[root@node01 conf]# mkdir -pv /Data/zookeeper mkdir: 已創建目錄 "/Data" mkdir: 已創建目錄 "/Data/zookeeper" [root@node01 conf]# echo "1" > /Data/zookeeper/myid #myid文件，里面的內容為數字，用於標識主機，如果這個文件沒有的話，zookeeper無法啟動

其他節點配置相同，除以下配置：

echo "x" > /Data/zookeeper/myid #唯一

3.啟動zookeeper（三台）

[root@node01 zookeeper]# cd /usr/local/zookeeper/bin [root@node01 bin]# ./zkServer.sh start ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Starting zookeeper ... taSTARTED [root@node01 bin]# tailf zookeeper.out 2019-02-13 14:05:28,088 [myid:] - INFO [main:QuorumPeerConfig@136] - Reading configuration from: /usr/local/zookeeper/bin/../conf/zoo.cfg 2019-02-13 14:05:28,102 [myid:] - INFO [main:QuorumPeer$QuorumServer@184] - Resolved hostname: 172.16.150.154 to address: /172.16.150.154 2019-02-13 14:05:28,102 [myid:] - INFO [main:QuorumPeer$QuorumServer@184] - Resolved hostname: 172.16.150.156 to address: /172.16.150.156 2019-02-13 14:05:28,103 [myid:] - INFO [main:QuorumPeer$QuorumServer@184] - Resolved hostname: 172.16.150.155 to address: /172.16.150.155 2019-02-13 14:05:28,103 [myid:] - INFO [main:QuorumPeerConfig@398] - Defaulting to majority quorums 2019-02-13 14:05:28,108 [myid:1] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3 2019-02-13 14:05:28,108 [myid:1] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0 2019-02-13 14:05:28,108 [myid:1] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled. 2019-02-13 14:05:28,119 [myid:1] - INFO [main:QuorumPeerMain@130] - Starting quorum peer 2019-02-13 14:05:28,128 [myid:1] - INFO [main:ServerCnxnFactory@117] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory 2019-02-13 14:05:28,134 [myid:1] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181 2019-02-13 14:05:28,144 [myid:1] - INFO [main:QuorumPeer@1158] - tickTime set to 2000 2019-02-13 14:05:28,144 [myid:1] - INFO [main:QuorumPeer@1204] - initLimit set to 10 2019-02-13 14:05:28,144 [myid:1] - INFO [main:QuorumPeer@1178] - minSessionTimeout set to -1 2019-02-13 14:05:28,144 [myid:1] - INFO [main:QuorumPeer@1189] - maxSessionTimeout set to -1 2019-02-13 14:05:28,151 [myid:1] - INFO [main:QuorumPeer@1467] - QuorumPeer communication is not secured! 2019-02-13 14:05:28,153 [myid:1] - INFO [main:QuorumPeer@1496] - quorum.cnxn.threads.size set to 20 2019-02-13 14:05:28,196 [myid:1] - INFO [ListenerThread:QuorumCnxManager$Listener@736] - My election bind port: /172.16.150.154:3888 ........

zookeeper服務檢查

[root@node01 bin]# netstat -nlpt | grep -E "2181|2888|3888" tcp 0 0 0.0.0.0:2181 0.0.0.0:* LISTEN 6242/java tcp 0 0 172.16.150.154:3888 0.0.0.0:* LISTEN 6242/java [root@node02 ~]# netstat -nlpt | grep -E "2181|2888|3888" tcp 0 0 0.0.0.0:2181 0.0.0.0:* LISTEN 5197/java tcp 0 0 172.16.150.155:3888 0.0.0.0:* LISTEN 5197/java [root@node03 ~]# netstat -nlpt | grep -E "2181|2888|3888" tcp 0 0 0.0.0.0:2181 0.0.0.0:* LISTEN 5304/java tcp 0 0 172.16.150.156:2888 0.0.0.0:* LISTEN 5304/java #哪台是leader，那么他就擁有2888端口，可以看到目前node3節點為leader
tcp 0 0 172.16.150.156:3888 0.0.0.0:* LISTEN 5304/java

測試服務器是否正常

[root@node01 bin]# yum install telnet nc -y [root@node01 bin]# telnet 172.16.150.154 2181 Trying 172.16.150.154... Connected to 172.16.150.154. Escape character is '^]'. exit Connection closed by foreign host. [root@node01 bin]# echo "stat"|nc 172.16.150.154 2181  #conf 可以顯示配置信息，cons可以顯示所有客戶端連接的詳細信息，mntr命令比stat命令更詳細 Zookeeper version: 3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 04:05 GMT Clients: /172.16.150.154:54989[0](queued=0,recved=1,sent=0) Latency min/avg/max: 0/0/0 Received: 1 Sent: 0 Connections: 1 Outstanding: 0 Zxid: 0x1000000d4 Mode: follower Node count: 138

連接zookeeper

[root@node01 bin]# ./zkCli.sh -server 172.16.150.154:2181 Connecting to 172.16.150.154:2181 2019-02-13 14:25:24,060 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 04:05 GMT .... [zk: 172.16.150.154:2181(CONNECTED) 0] h  #查看命令幫助 ZooKeeper -server host:port cmd args stat path [watch] set path data [version] ls path [watch] delquota [-n|-b] path ls2 path [watch] setAcl path acl setquota -n|-b val path history redo cmdno printwatches on|off delete path [version] sync path listquota path rmr path get path [watch] create [-s] [-e] path data acl addauth scheme auth quit getAcl path close connect host:port [zk: 172.16.150.154:2181(CONNECTED) 1] quit #退出

設置jconsole連接zookeeper 

[root@node01 bin]# vim zkServer.sh #修改54行，172.16.150.154是本機的ip地址，8899是jconsole的連接地址，關閉ssl和認證 ZOOMAIN="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=$JMXLOCALONLY -Djava.rmi.server.hostname=172.16.150.154 -Dcom.sun.management.jmxremote.port=8899 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false org.apache .zookeeper.server.quorum.QuorumPeerMain"

./zkServer.sh stop && ./zkServer.sh start #重啟服務，使用jconsole連接zookeeper服務器，選擇遠程連接，輸入172.16.150.154:8899 即可

#登錄jconsole

 

zookeeper開啟超級用戶   #關於zookeeper ACL權限請參考官方文檔

當設置了znode權限，但是密碼忘記了怎么辦？如果忘記了該子節點的授權用戶名還有密碼。這里是比較蛋疼的事情。由於我們基本上找不到because在base64反編碼后再sha1反編碼后的樣子，所以基本上這個節點的控制權可以說是失去了。還好Zookeeper提供了超級管理員機制。

[root@node01 bin]# cd /usr/local/zookeeper/lib/ [root@node01 lib]# java -cp ../zookeeper-3.4.13.jar:./log4j-1.2.17.jar:./slf4j-api-1.7.25.jar:./slf4j-log4j12-1.7.25.jar org.apache.zookeeper.server.auth.DigestAuthenticationProvider super:super super:super->super:gG7s8t3oDEtIqF6DM9LlI/R+9Ss=  #生成密文
[root@node01 lib]# vim ../bin/zkServer.sh SUPER_ACL="-Dzookeeper.DigestAuthenticationProvider.superDigest=super:gG7s8t3oDEtIqF6DM9LlI/R+9Ss="

#添加以上標記的內容

驗證用戶是否有效

[root@node01 lib]# cd ../bin/ [root@node01 bin]# ./zkServer.sh stop #修改配置文件后重啟服務 ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Stopping zookeeper ... STOPPED [root@node01 bin]# ./zkServer.sh start ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [root@node01 bin]# ./zkCli.sh -server 172.16.150.154 ...... [zk: 172.16.150.154(CONNECTED) 0] addauth digest super:super #對之前添加的用戶進行認證 [zk: 172.16.150.154(CONNECTED) 1] quit

三、kafka集群安裝

kafka同樣依賴Java環境，由於和zookeeper在相同的機器上，之前已經安裝過了，所有可以直接跳過Java環境安裝

1.安裝kafka

[root@node01 ~]#wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.0.1/kafka_2.11-2.0.1.tgz [root@node01 ~]#tar xf kafka_2.11-2.0.1.tgz -C /usr/local [root@node01 ~]#cd /usr/local [root@node01 local]# ln -sv kafka_2.11-2.0.1 kafka [root@node01 local]# cd kafka/config/ [root@node01 config]#cp server.properties server.properties-bak [root@node01 config]# grep "^[a-Z]" server.properties broker.id=1 #唯一 listeners=PLAINTEXT://172.16.150.154:9092 #修改為本機地址 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/Data/kafka-logs #數據目錄，kafka-logs會自動采集 num.partitions=3 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=172.16.150.154:2181,172.16.150.155:2181,172.16.150.156:2181 #zokeeper集群地址，以","為分割 zookeeper.connection.timeout.ms=6000 group.initial.rebalance.delay.ms=0

其他節點配置相同，除以下內容：

broker.id=1 #唯一 listeners=PLAINTEXT://172.16.150.154:9092 #修改為本機地址

啟動服務

[root@node01 config]# cd ../bin [root@node01 bin]# ./kafka-server-start.sh -daemon ../config/server.properties #后台運行

驗證服務是否正常

登錄zookeeper驗證：

[zk: 172.16.150.154(CONNECTED) 5] get /brokers/ids/1 #查看節點broker id為1的信息 {"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://172.16.150.154:9092"],"jmx_port":-1,"host":"172.16.150.154","timestamp":"1549953312989","port":9092,"version":4} cZxid = 0x10000002e ctime = Tue Feb 12 14:35:13 CST 2019 mZxid = 0x10000002e mtime = Tue Feb 12 14:35:13 CST 2019 pZxid = 0x10000002e cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x10077feb7bc0001 dataLength = 198 numChildren = 0

創建topic驗證

#154上創建一個生產者
[root@node01 ~]# cd /usr/local/kafka/bin/
[root@node01 bin]# ./kafka-topics.sh --create --zookeeper 172.16.150.154:2181 --replication-factor 1 --partitions 1 --topic Test Created topic "Test". [root@node01 bin]# ./kafka-console-producer.sh --broker-list 172.16.150.154:9092 --topic Test #其他服務器上創建一個消費者
[root@node02 ~]# cd /usr/local/kafka/bin/ [root@node02 bin]# ./kafka-console-consumer.sh --bootstrap-server 172.16.150.155:9092 --topic Test --from-beginning
#啟動成功后，在154上輸入任意內容，在另一台機器上查看是否會同步顯示

四、zookeeper及kafka監控工具

1.zookeeper監控工具(沒有安裝過，有需要請參考官方文檔)

zookeeper監控工具地址：https://github.com/soabase/exhibitor

2.kafka監控工具

1)KafkaOffsetMonitor

[root@node01 ~]#mkdir KafkaMonitor [root@node01 ~]#cd KafkaMonitor/ [root@node01 ~]#wget https://github.com/quantifind/KafkaOffsetMonitor/releases/download/v0.2.1/KafkaOffsetMonitor-assembly-0.2.1.jar [root@node01 ~]#nohup java -cp KafkaOffsetMonitor-assembly-0.2.0.jar com.quantifind.kafka.offsetapp.OffsetGetterWeb --zk 172.16.150.154:2181,172.16.150.155:2181,172.16.150.156:2181 -port 8088 --refresh 5.seconds --retain 1.days & 

訪問(由於測試環境沒有數據，所有我直接登錄生產環境來演示)：

查看曾經消費者的情況

查看其中任意一個消費者信息

主意lag字段，表示是否有延遲

查看topic

 

2)kafka-manager

[root@node01 ~]# unzip kafka-manager-1.3.3.7.zip #直接使用已經編譯完成的軟件包(鏈接: https://pan.baidu.com/s/12sswyPo7-e9R3mZQ3ba-dA 提取碼: jz6s) [root@node01 ~]# cd kafka-manager-1.3.3.7 [root@node01 ~]# cd conf/ [root@node01 ~]# vim application.conf kafka-manager.zkhosts="172.16.150.154:2181,172.16.150.155:2181,172.16.150.156:2181" #填寫zookeeper服務器地址和端口 [root@node01 ~]#cd ../bin/ [root@node01 ~]# ./kafka-manager -Dconfig.file=../conf/application.conf -Dhttp.port=8888 #8888表示監聽端口，啟動后直接訪問

#kafka-manager安裝需要編譯，並且過程復雜、成功率低建議使用其他人已經編譯過得直接使用

3)kafka eagle

未實驗，聽說不錯有興趣的可以學習一下

作為一個初學者，有很多地方都沒有理解，寫的比較簡單，望海涵！

 

參考文檔：

https://zookeeper.apache.org/doc/r3.4.13/zookeeperAdmin.html

https://blog.csdn.net/pdw2009/article/details/73794525

https://blog.csdn.net/lizhitao/article/details/25667831

https://www.cnblogs.com/dadonggg/p/8242682.html

https://www.cnblogs.com/dadonggg/p/8205302.html