安裝zookeeper 3.4.11單機集群
1.說明
需要說明的幾個問題：1）kafka依賴zookeeper,請先安裝zookeeper。2）zookeeper依賴java環境，請先安裝jdk。3）zookeeper單機集群規划：3個節點，端口號分別是2181、2812、2813。4）zookeeper單機集群關鍵在於隔離各節點的端口、數據。
2.zookeeper單機集群安裝
1）zookeeper安裝目錄
/opt/app/zookeeper，如果沒有，就創建有關目錄文件。
2）zookeeper集群節點規划
/opt/app/zookeeper下3個節點的子文件夾是：2181,2182,2183，各個節點的數據文件夾data，數據日志文件夾datalogs。
3）安裝zookeeper集群
安裝zookeeper具體步驟如下：
安裝目錄：/opt/app/zookeeper，如果沒有，就創建有關目錄文件,我的虛擬機沒有，所以我需要創建。
mkdir /opt/app/zookeeper -p
下載並且解壓zookeeper安裝
[root@elk01 software]# pwd
/opt/software
[root@elk01 software]# ll
total 84128
-rw-r--r--. 1 root root 49475271 Nov 1 2017 kafka_2.11-1.0.0.tgz
-rw-r--r--. 1 root root 36668066 Nov 9 2017 zookeeper-3.4.11.tar.gz
[root@elk01 software]# tar -zxvf zookeeper-3.4.11.tar.gz -C /opt/app/
創建zookeeper有關文件夾結構
[root@elk01 app]# pwd
/opt/app
[root@elk01 app]# ll
total 4
drwxr-xr-x 2 root root 6 Mar 14 09:44 zookeeper
drwxr-xr-x 10 502 games 4096 Nov 2 2017 zookeeper-3.4.11
[root@elk01 app]# mkdir -p zookeeper/218{1,2,3}/{data,datalogs}
復制zookeeper程序文件到各個節點
[root@elk01 app]# pwd
/opt/app
You have new mail in /var/spool/mail/root
[root@elk01 app]# ll
total 4
drwxr-xr-x 5 root root 42 Mar 14 09:49 zookeeper
drwxr-xr-x 10 502 games 4096 Nov 2 2017 zookeeper-3.4.11
[root@elk01 app]# cp -r zookeeper-3.4.11 zookeeper/2181
[root@elk01 app]# cp -r zookeeper-3.4.11 zookeeper/2182
[root@elk01 app]# cp -r zookeeper-3.4.11 zookeeper/2183
原來的zookeeper文件夾可以刪除
[root@elk01 app]# rm -rf zookeeper-3.4.11/
[root@elk01 app]# pwd
/opt/app
[root@elk01 app]# ll
total 0
drwxr-xr-x 5 root root 42 Mar 14 09:49 zookeeper
修改zookeeper zoo.cfg配置
zoo.cfg是zookeeper的配置文件，放在各個節點的zookeeper-3.4.11/conf/文件夾下。可以復制zoo_sample.cfg配置模板。
配置節點1：
[root@elk01 app]# cd zookeeper/2181/zookeeper-3.4.11/conf/
[root@elk01 conf]# cp zoo_sample.cfg zoo.cfg
[root@elk01 conf]# grep "^[a-z]" zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/app/zookeeper/2181/data
clientPort=2181
dataLogDir=/opt/app/zookeeper/2181/datalogs
server.1=10.96.211.209:2881:3881
server.2=10.96.211.209:2882:3882
server.3=10.96.211.209:2883:3883
同理，節點2和節點3也這樣配置。注意修改clientPort、dataDir、dataLogDir。
配置節點2：
[root@elk01 conf]# pwd
/opt/app/zookeeper/2182/zookeeper-3.4.11/conf
[root@elk01 conf]# cp zoo_sample.cfg zoo.cfg
[root@elk01 conf]# pwd
/opt/app/zookeeper/2182/zookeeper-3.4.11/conf
[root@elk01 conf]# grep "^[a-z]" zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/app/zookeeper/2182/data
clientPort=2182
dataLogDir=/opt/app/zookeeper/2182/datalogs
server.1=10.96.211.209:2881:3881
server.2=10.96.211.209:2882:3882
server.3=10.96.211.209:2883:3883
配置節點3：
[root@elk01 conf]# pwd
/opt/app/zookeeper/2183/zookeeper-3.4.11/conf
[root@elk01 conf]# cp zoo_sample.cfg zoo.cfg
[root@elk01 conf]# grep "^[a-z]" zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/app/zookeeper/2183/data
clientPort=2183
dataLogDir=/opt/app/zookeeper/2183/datalogs
server.1=10.96.211.209:2881:3881
server.2=10.96.211.209:2882:3882
server.3=10.96.211.209:2883:3883
zookeeper配置說明：
clientPort：客戶端連接端口
server.x：集群中的節點（包括自己），x對應myid的數字，2個端口分別用於集群選舉通訊
myid修改配置
每個節點必須有myid配置文件，記錄節點的唯一標識，必須放在dataDir文件夾下
id值必須與上面配置的server.x中的x對應。
[root@elk01 zookeeper]# pwd
/opt/app/zookeeper
[root@elk01 zookeeper]# ll
total 0
drwxr-xr-x 5 root root 58 Mar 14 09:59 2181
drwxr-xr-x 5 root root 58 Mar 14 09:59 2182
drwxr-xr-x 5 root root 58 Mar 14 09:59 2183
[root@elk01 zookeeper]# touch 2181/data/myid && echo "1" > 2181/data/myid
[root@elk01 zookeeper]# touch 2182/data/myid && echo "2" > 2182/data/myid
[root@elk01 zookeeper]# touch 2183/data/myid && echo "3" > 2183/data/myid
查看myid文件內容
[root@elk01 zookeeper]# more 218*/data/myid
::::::::::::::
2181/data/myid
::::::::::::::
1
::::::::::::::
2182/data/myid
::::::::::::::
2
::::::::::::::
2183/data/myid
::::::::::::::
3
如果配置了iptables防火牆，需要開啟有關端口，我自己虛擬機防火牆關了，所以不需要配置。
4）啟動zookeeper集群
啟動各個節點zookeeper
進入各個節點啟動zookeeper，是為了讓運行日志zookeeper.out生成在各個節點目錄下。
[root@elk01 2181]# pwd
/opt/app/zookeeper/2181
[root@elk01 2181]# ll
total 4
drwxr-xr-x 2 root root 18 Mar 14 10:21 data
drwxr-xr-x 2 root root 6 Mar 14 09:49 datalogs
drwxr-xr-x 10 root root 4096 Mar 14 09:59 zookeeper-3.4.11
[root@elk01 2181]# zookeeper-3.4.11/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@elk01 2181]# cd ../2182/
[root@elk01 2182]# zookeeper-3.4.11/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/2182/zookeeper-3.4.11/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@elk01 2182]# cd ../2183
[root@elk01 2183]# zookeeper-3.4.11/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/2183/zookeeper-3.4.11/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
5）查看zookeeper集群狀態
節點1：
[root@elk01 zookeeper]# pwd
/opt/app/zookeeper
[root@elk01 zookeeper]# ll
total 0
drwxr-xr-x 5 root root 79 Mar 14 10:26 2181
drwxr-xr-x 5 root root 79 Mar 14 10:26 2182
drwxr-xr-x 5 root root 79 Mar 14 10:26 2183
[root@elk01 zookeeper]# 2181/zookeeper-3.4.11/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../conf/zoo.cfg
Mode: follower
節點2：
[root@elk01 zookeeper]# 2182/zookeeper-3.4.11/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/2182/zookeeper-3.4.11/bin/../conf/zoo.cfg
Mode: leader
節點3：
[root@elk01 zookeeper]# 2183/zookeeper-3.4.11/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/2183/zookeeper-3.4.11/bin/../conf/zoo.cfg
Mode: follower
如上，leader表示主節點，follower表示從節點。目前狀態是1主2從，狀態正常！
6）客戶端連接zookeeper集群測試
如下表示連接正常，zookeeper單機集群搭建大功告成！
[root@elk01 zookeeper]# pwd
/opt/app/zookeeper
[root@elk01 zookeeper]# 2181/zookeeper-3.4.11/bin/zkCli.sh
Connecting to localhost:2181
2022-03-14 10:32:25,115 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0, built on 11/01/2017 18:06 GMT
2022-03-14 10:32:25,117 [myid:] - INFO [main:Environment@100] - Client environment:host.name=elk01
2022-03-14 10:32:25,117 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_312
2022-03-14 10:32:25,118 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Red Hat, Inc.
2022-03-14 10:32:25,118 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre
2022-03-14 10:32:25,118 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../build/classes:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../build/lib/*.jar:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../lib/slf4j-api-1.6.1.jar:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../lib/netty-3.10.5.Final.jar:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../lib/log4j-1.2.16.jar:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../lib/jline-0.9.94.jar:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../lib/audience-annotations-0.5.0.jar:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../zookeeper-3.4.11.jar:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../src/java/lib/*.jar:/opt/app/zookeeper/2181/zookeeper-3.4.11/bin/../conf:
2022-03-14 10:32:25,118 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2022-03-14 10:32:25,118 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2022-03-14 10:32:25,118 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2022-03-14 10:32:25,119 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2022-03-14 10:32:25,119 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2022-03-14 10:32:25,119 [myid:] - INFO [main:Environment@100] - Client environment:os.version=3.10.0-1160.el7.x86_64
2022-03-14 10:32:25,119 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root
2022-03-14 10:32:25,119 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root
2022-03-14 10:32:25,119 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/opt/app/zookeeper
2022-03-14 10:32:25,119 [myid:] - INFO [main:ZooKeeper@441] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@68de145
Welcome to ZooKeeper!
2022-03-14 10:32:25,132 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1035] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2022-03-14 10:32:25,179 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@877] - Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
[zk: localhost:2181(CONNECTING) 0] 2022-03-14 10:32:25,208 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1302] - Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x1000041f3ec0000, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0]
安裝kafka 2.11-1.0.0單機集群
1.說明
需要說明幾個問題:
2.Kafka單機集群安裝
1）單機Kafka集群安裝和配置
Kafka單機集群具體部署和配置步驟如下：
安裝目錄：/opt/app/kafka，如果沒有，就創建。
mkdir /opt/app/kafka -p
[root@elk01 software]# pwd
/opt/software
下載並且解壓kafka安裝包
[root@elk01 software]# tar -zxvf kafka_2.11-1.0.0.tgz
創建kafka有關目錄結構
[root@elk01 app]# pwd
/opt/app
[root@elk01 app]# ll
total 0
drwxr-xr-x 2 root root 6 Mar 14 11:05 kafka
drwxr-xr-x 5 root root 42 Mar 14 09:49 zookeeper
[root@elk01 app]# mkdir -p kafka/909{2,3,4}/logs
復制kafka安裝程序文件到各個節點目錄下
[root@elk01 software]# pwd
/opt/software
[root@elk01 software]# ll
total 84128
drwxr-xr-x 6 root root 89 Oct 28 2017 kafka_2.11-1.0.0
-rw-r--r--. 1 root root 49475271 Nov 1 2017 kafka_2.11-1.0.0.tgz
-rw-r--r--. 1 root root 36668066 Nov 9 2017 zookeeper-3.4.11.tar.gz
[root@elk01 software]# cp -r kafka_2.11-1.0.0 /opt/app/kafka/9092/
[root@elk01 software]# cp -r kafka_2.11-1.0.0 /opt/app/kafka/9093/
You have new mail in /var/spool/mail/root
[root@elk01 software]# cp -r kafka_2.11-1.0.0 /opt/app/kafka/9094/
修改kafka集群配置
修改每個節點下kafka_2.11-1.0.0/config/server.properties配置文件。
需要修改的配置項修改后如下：
節點1：
[root@elk01 kafka]# pwd
/opt/app/kafka
[root@elk01 kafka]# vim 9092/kafka_2.11-1.0.0/config/server.properties
broker.id=1
listeners=PLAINTEXT:10.96.211.209:9092
advertised.listeners=PLAINTEXT://10.96.211.209:9092
log.dirs=/opt/app/kafka/9092/logs
zookeeper.connect=10.96.211.209:2181,10.96.211.209:2182,10.96.211.209:2183
節點2：
root@elk01 kafka]# pwd
/opt/app/kafka
[root@elk01 kafka]# vim 9093/kafka_2.11-1.0.0/config/server.properties
broker.id=2
listeners=PLAINTEXT:10.96.211.209:9093
advertised.listeners=PLAINTEXT://10.96.211.209:9093
log.dirs=/opt/app/kafka/9093/logs
zookeeper.connect=10.96.211.209:2181,10.96.211.209:2182,10.96.211.209:2183
節點3：
root@elk01 kafka]# pwd
/opt/app/kafka
[root@elk01 kafka]# vim 9094/kafka_2.11-1.0.0/config/server.properties
broker.id=3
listeners=PLAINTEXT://10.96.211.209:9094
advertised.listeners=PLAINTEXT://10.96.211.209:9094
log.dirs=/opt/app/kafka/9094/logs
zookeeper.connect=10.96.211.209:2181,10.96.211.209:2182,10.96.211.209:2183
kafka consumer.properties配置文件修改
每個節點的consumer.properties只需要修改如下配置，需要修改的配置項修改后如下：
[root@elk01 kafka]# pwd
/opt/app/kafka
[root@elk01 kafka]# ll
total 0
drwxr-xr-x 4 root root 42 Mar 14 11:10 9092
drwxr-xr-x 4 root root 42 Mar 14 11:10 9093
drwxr-xr-x 4 root root 42 Mar 14 11:10 9094
vim 9092/kafka_2.11-1.0.0/config/consumer.properties
bootstrap.servers=10.96.211.209:9092,10.96.211.209:9093,10.96.211.209:9094
vim 9093/kafka_2.11-1.0.0/config/consumer.properties
bootstrap.servers=10.96.211.209:9092,10.96.211.209:9093,10.96.211.209:9094
vim 9094/kafka_2.11-1.0.0/config/consumer.properties
bootstrap.servers=10.96.211.209:9092,10.96.211.209:9093,10.96.211.209:9094
如果開啟了防火牆，需要開啟有關端口，我的虛擬機防火牆關了，所以不需要配置。
2）啟動Kafka集群
節點1：
[root@elk01 bin]# pwd
/opt/app/kafka/9092/kafka_2.11-1.0.0/bin
[root@elk01 bin]# ./kafka-server-start.sh -daemon ../config/server.properties
節點2：
[root@elk01 bin]# cd /opt/app/kafka/9093/kafka_2.11-1.0.0/bin
[root@elk01 bin]# ./kafka-server-start.sh -daemon ../config/server.properties
節點3：
[root@elk01 bin]# cd /opt/app/kafka/9094/kafka_2.11-1.0.0/bin
[root@elk01 bin]# ./kafka-server-start.sh -daemon ../config/server.properties
3）測試Kafka集群是否可用
1）創建一個test主題（topic）
[root@elk01 bin]# pwd
/opt/app/kafka/9092/kafka_2.11-1.0.0/bin
[root@elk01 bin]# ./kafka-topics.sh --create --zookeeper 10.96.211.209:2181,10.96.211.209:2182,10.96.211.209:2183 --replication-factor 3 --partitions 3 --topic 0314rontest
Created topic "0314rontest".
2）連接生產者，並發送數據
[root@elk01 bin]# pwd
/opt/app/kafka/9092/kafka_2.11-1.0.0/bin
[root@elk01 bin]# ./kafka-console-producer.sh --broker-list 10.96.211.209:9092,10.96.211.209:9093,10.96.211.209:9094 --topic 0314rontest
>nihao
>haha
>hehe
>
3)新開一個ssh會話，連接消費者
如上，消費者收到消息，說明kafka集群是OK的！測試成功，kafka集群搭建大功告成！