Centos7.4 kafka集群安裝與kafka-eagle1.3.9的安裝
集群規划:
| hostname | Zookeeper | Kafka | kafka-eagle |
|---|---|---|---|
| kafka01 | √ | √ | √ |
| kafka02 | √ | √ | |
| kafka03 | √ | √ |
准備工作:
-
網絡配置
vim /etc/sysconfig/network-scripts/ifcfg-ens33# kafka01 TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=static DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens33 DEVICE=ens33 ONBOOT=yes IPADDR=192.168.180.200 # 你的虛擬網卡VMNet8, 如果是橋接的改成橋接網卡VMNet1 PREFIX=24 GATEWAY=192.168.180.2 DNS1=114.114.114.114 IPV6_PRIVACY=no# kafka02 TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=static DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens33 DEVICE=ens33 ONBOOT=yes IPADDR=192.168.180.201 PREFIX=24 GATEWAY=192.168.180.2 DNS1=114.114.114.114 IPV6_PRIVACY=no# kafka03 TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=static DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens33 DEVICE=ens33 ONBOOT=yes IPADDR=192.168.180.200 PREFIX=24 GATEWAY=192.168.180.2 DNS1=114.114.114.114 IPV6_PRIVACY=no -
復制三台虛擬機
-
修改主機名及hosts
vim /etc/hostname- 將三台主機的主機名分別修改為kafka01, kafka02, kafka03
vim /etc/hosts修改為:
127.0.0.1 localhost 192.168.180.200 kafka01 192.168.180.201 kafka02 192.168.180.202 kafka03 -
關閉防火牆
# ps: 這里就不搞iptables防火牆了, 直接關, 不然開得一個個開放端口, 比較煩
service firewalld stop
-
免密登錄
ssh-keygen # 然后enter, enter, enter 直到生成RSA, 默認會在當前用戶的家目錄下生成.ssh [root@kafka02 ~]# ls -ah . .bash_history .bashrc .cshrc .pki .viminfo .xauthEJoei0 .. .bash_logout .cache .dbus .ssh .xauthBayboq anaconda-ks.cfg .bash_profile .config initial-setup-ks.cfg .tcshrc .xauthcwYAMX.ssh隱藏目錄中有id_rsa(私鑰) 和 id_rsa.pub(公鑰)
cd ~/.ssh 創建authorized_keys，將三台虛擬機的公鑰都存入其中:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCZJgAqbq+/peLIdV5ZpDSS3+8nYJSg/2ZON8rj9WmXlefGLgoegRFExCN4LPEN72/QEl/ZOBxtc00Tev+pSM30veOkxJTc+XXGu5LlFp3l5bbB0nIZG5hYKstT7FYPDfNqZkJrNp+6l0Yl/Eihv9R4LPT5K2A4TbxtFi7s9c0n2fUp5XOLjQWU9qj1x7bajV0nLjV3gOeejqNKVywr2xQqOJX7DQ/e1jlKekvUGHpa/fWVyTc/rK7dCvyddAqqrO5P+7Tu9CqLvcvV1hFtgVechCJ0VCw3tId3qczJzEj51KJb1hErzCFz+7azOMkLhw0/NkX36PjgPX6F59AOsPhX root@kafka01 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8XpLYRdcuaxXUOOW+OtfYYXmH8TS1ETd22A5898JCZgIIqiZjrqMoJCARxcAkkaLjJu4x6XaYA470i9mYy2lf1N2XWI2zxt1Fne8G22ZTtCeY7Qm1obVcPM3WqUNN/EvOXT9qDuitQnCEhS3rvnzp/h3ZuYgLv3r7XdDDVRqMxc4FdAIkGjJCu68lpvDeXHY1nBzcpoyAnXy8WhAnSmP8RSTh1XbDhwvgvcCmnQbGiaNHSGNTjYMU2a8mOEenoF4osB+Mys5WkNaJ0RYU39vU2RJdQb/YxGbbQdTszTz1Y0aRceONDr3Wpo2mCf9whvKpT7d72FFV9iIyPy0ax8sv root@kafka02 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCft3A+poCNF6N//3hADQnaRgGDnPe/58FhSJNPPx64AermLE83Nj0E9tu5CD6Z30FE5LiRAAkLb1yXBIxtU4g5X88TQ8CfxNqE9ZkhAzeaaBXGJ8Kh9/HAAP5J63YCwXIMOFjJpA1RJ2QPxWC31jF+qLY33XSb5b643kGp0QX0m1FVjoh9y3ScTIF2Kznl5atzPG7j+100AQjBztgyZU1AdXqIzdnD6+vKTHYC1lxIkp2PWleMsN+pWlV6JFhhYZzVnKEsOtW9SDDgsUbKPr1y3ofowSTYvuZx8xSkk3gSXb6HUSedc92EiFrFoONazr+is7y3ZPdspxuN7NtISIwh root@kafka03每台虛擬機的authorized_keys文件一致
免密登錄測試:
[root@kafka03 bin]# ssh root@192.168.180.201 Last login: Fri Oct 4 13:06:53 2019 [root@kafka02 ~]#ps： 第一次會提示是否將該秘鑰保存, 保存后.ssh文件目錄下會生成一個known_hosts，里面存有已知的主機sha(Security HashCode Algorithum)信息:
比如我的:
192.168.180.200 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOozduEWhzGQL5NzN1ZTpRUqfXQBNCDEq4YHeElcxuk17xsGr6job8H5lxFkf+0FwcD2ZFkbZeUJS/UnR0EtdzA= 192.168.180.201 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOozduEWhzGQL5NzN1ZTpRUqfXQBNCDEq4YHeElcxuk17xsGr6job8H5lxFkf+0FwcD2ZFkbZeUJS/UnR0EtdzA= 192.168.180.202 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOozduEWhzGQL5NzN1ZTpRUqfXQBNCDEq4YHeElcxuk17xsGr6job8H5lxFkf+0FwcD2ZFkbZeUJS/UnR0EtdzA=
JDK安裝
-
將jdk的tar包上傳，然后tar -zxvf 該包 -C 想要解壓到的目錄
-
我這邊是解壓到了/opt/ronnie/目錄, 並且 mv jdk1.8XXXX jdk1.8 將它改名為了jdk1.8
[root@kafka02 ronnie]# ll
總用量 0
drwxr-xr-x. 7 10 143 245 12月 16 2018 jdk1.8
drwxrwxrwx. 8 root root 120 10月 4 13:00 kafka
drwxr-xr-x. 8 root root 158 10月 2 20:20 zookeeper
-
修改系統配置文件
vim /etc/profile # 添加jdk路徑 export JAVA_HOME=/opt/ronnie/jdk1.8 export PATH=$JAVA_HOME/bin:$PATH檢查配置是否成功:
[root@kafka02 .ssh]# java -version java version "1.8.0_202" Java(TM) SE Runtime Environment (build 1.8.0_202-b08) Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed mode) [root@kafka02 .ssh]# javac -version javac 1.8.0_202將配置好的jdk目錄傳送給其他幾台機器:
scp -r /opt/ronnie/jdk1.8 root@192.168.180.xxx:/opt/ronnie
Zookeeper安裝
-
上傳Zookeeper的tar包, 我這里用的是3.5.5
-
tar -zxvf 該包 -C 你想要解壓到的目錄
我這里一樣解壓到了/opt/ronnie/下, 並 mv zookeeperXXXXX zookeeper 將其重命名為zookeeper
- 創建data目錄:
mkdir -p /opt/ronnie/zookeeper/data- 在data目錄下創建日志目錄:
mkdir -p /opt/ronnie/zookeeper/data/log- 在data目錄下創建zk目錄:
mkdir -p /opt/ronnie/zookeeper/data/zk-
cd zk/, 創建myid, 三台虛擬機kafka01, kafka02, kafka03 分別對應myid 1,2,3
-
進入zookeeper的配置文件目錄, 創建並修改zoo.cfg
cd /opt/ronnie/zookeeper/conf # 拷貝一份zoo_sample.cfg 為 zoo.cfg cp zoo_sample.cfg zoo.cfg # 修改zoo.cfg文件 vim zoo.cfgzoo.cfg:
# The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/opt/ronnie/zookeeper/data/zk dataLogDir=/opt/ronnie/zookeeper/data/log # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to "0" to disable auto purge feature #autopurge.purgeInterval=1 server.1=kafka01:2888:3888 server.2=kafka02:2888:3888 server.3=kafka03:2888:3888 -
將zookeeper添加至環境目錄
vim /etc/profile # 添加Zookeeper配置 export ZOOKEEPER_HOME=/opt/ronnie/zookeeper export PATH=$ZOOKEEPER_HOME/bin:$PATH -
將配置好的zookeeper目錄傳送給其他幾台虛擬機
scp -r /opt/ronnie/zookeeper root@192.168.180.xxx:/opt/ronnie-
啟動Zookeeper: zkServer.sh start
-
檢查Zookeeper啟動狀態: zkServer.sh status
成功啟動案例:
[root@kafka02 conf]# zkServer.sh status ZooKeeper JMX enabled by default Using config: /opt/ronnie/zookeeper/bin/../conf/zoo.cfg Client port found: 2181. Client address: localhost. Mode: leader 如果報了: JMX enabled by default
Using config: /hadoop/zookeeper/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.可能是配置文件的問題, 也可能是防火牆的問題, 可以cd logs 去查看下日志中的報錯來定位問題
Kafka安裝
- 上傳kafka的tar包, 並tar -zxvf -C 你想要解壓的目錄,
我這用的是2.12-2.3.0版本的kafka
-
修改配置文件
vim /opt/ronnie/kafka/config/server.properties # server.properties 服務器屬性 # Licensed to the Apache Software Foundation (ASF) under one or more # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # see kafka.server.KafkaConfig for additional details and defaults ############################# Server Basics ############################# # The id of the broker. This must be set to a unique integer for each broker. # brokerId 需要是唯一的, 這里我就把每台的brokerId分別設置成200,201 和 202 了 broker.id=200 # port 端口 port=9092 # hostname 主機名, 分別修改為各自對應的 host.name=kafka01 ############################# Socket Server Settings ############################# # The address the socket server listens on. It will get the value returned from # java.net.InetAddress.getCanonicalHostName() if not configured. # FORMAT: # listeners = listener_name://host_name:port # EXAMPLE: # listeners = PLAINTEXT://your.host.name:9092 # 監聽者也需要將主機名設置為各自對應的 listeners=PLAINTEXT://kafka01:9092 #advertised.listeners=PLAINTEXT://your.host.name:9092 #listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL num.network.threads=3 socket.send.buffer.bytes=102400 # The receive buffer (SO_RCVBUF) used by the socket server socket.receive.buffer.bytes=102400 # The maximum size of a request that the socket server will accept (protection against OOM) socket.request.max.bytes=104857600 ############################# Log Basics ############################# # A comma separated list of directories under which to store log files log.dirs=/tmp/kafka-logs # The default number of log partitions per topic. More partitions allow greater # parallelism for consumption, but this will also result in more files across # the brokers. num.partitions=1 # The number of threads per data directory to be used for log recovery at startup and flushing at shutdown. # This value is recommended to be increased for installations with data dirs located in RAID array. num.recovery.threads.per.data.dir=1 ############################# Internal Topic Settings ############################# offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 ############################# Log Flush Policy ############################# # Messages are immediately written to the filesystem but by default we only fsync() to sync # the OS cache lazily. The following configurations control the flush of data to disk. # There are a few important trade-offs here: # 1. Durability: Unflushed data may be lost if you are not using replication. # The settings below allow one to configure the flush policy to flush data after a period of time or # every N messages (or both). This can be done globally and overridden on a per-topic basis. #log.flush.interval.ms=1000 # be set to delete segments after a period of time, or after a given size has accumulated. # A segment will be deleted whenever *either* of these criteria are met. Deletion always happens # from the end of the log. # The minimum age of a log file to be eligible for deletion due to age log.retention.hours=168 # A size-based retention policy for logs. Segments are pruned from the log unless the remaining # segments drop below log.retention.bytes. Functions independently of log.retention.hours. #log.retention.bytes=1073741824 # The maximum size of a log segment file. When this size is reached a new log segment will be created. log.segment.bytes=1073741824 # The interval at which log segments are checked to see if they can be deleted according # to the retention policies log.retention.check.interval.ms=300000 ############################# Zookeeper ############################# # Zookeeper connection string (see zookeeper docs for details). # This is a comma separated host:port pairs, each corresponding to a zk # server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002". # You can also append an optional chroot string to the urls to specify the # root directory for all kafka znodes. # zookeeper的連接 zookeeper.connect=kafka01:2181,kafka02:2181,kafka03:2181 # Timeout in ms for connecting to zookeeper zookeeper.connection.timeout.ms=6000 ############################# Group Coordinator Settings ############################# # The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance. # The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms. # The default value for this is 3 seconds. # We override this to 0 here as it makes for a better out-of-the-box experience for development and testing. # However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup. group.initial.rebalance.delay.ms=0- 將配置好的kafka目錄發送給其他虛擬機
scp -r /opt/ronnie/kafka root@192.168.180.xxx:/opt/ronnie- 在三台虛擬機上啟動kafka
cd /opt/ronnie/kafka bin/kafka-server-start.sh config/server.properties&- 檢查kafka是否啟動, 成功案例
[root@kafka02 ronnie]# jps 15857 Jps 11672 QuorumPeerMain 12569 Kafka
Kafka-eagle安裝
- 下載
https://codeload.github.com/smartloli/kafka-eagle-bin/tar.gz/v1.3.9
- 上傳tar包並 tar -zxvf -C /opt/ronnie/kafka， 然后 mv kafka-eagle-xxxxx kafka-eagle 將解壓后的文件重命名為kafka-eagle
- 然后你會發現里面還有一個tar包..... tar -zxvf 該包 解壓到當前目錄並刪除tar包
- 配置環境變量
vim /etc/profile
# profile
export KE_HOME=/opt/ronnie/kafka/kafka-eagle/kafka-eagle-web-1.3.9
export PATH=$KE_HOME/bin:$PATH
-
修改eagle設置
vim /opt/ronnie/kafka/kafka-eagle/kafka-eagle-web-1.3.9/conf/system-config.properties###################################### # multi zookeeper&kafka cluster list ###################################### # 設置集群名和zookeeper集群列表 kafka.eagle.zk.cluster.alias=cluster1 cluster1.zk.list=kafka01:2181,kafka02:2181,kafka03:2181 # cluster2.zk.list=xdn10:2181,xdn11:2181,xdn12:2181 ###################################### # zk client thread limit ###################################### kafka.zk.limit.size=25 ###################################### # kafka eagle webui port ###################################### kafka.eagle.webui.port=8048 ###################################### # kafka offset storage ###################################### cluster1.kafka.eagle.offset.storage=kafka #cluster2.kafka.eagle.offset.storage=zk ###################################### # enable kafka metrics ###################################### # 是否使用圖表, 使用的話需要創建一個JMX端口 kafka.eagle.metrics.charts=true ###################################### # kafka sql topic records max ###################################### kafka.eagle.sql.topic.records.max=5000 # alarm email configure ###################################### kafka.eagle.mail.enable=true kafka.eagle.mail.sa=alert_sa@163.com kafka.eagle.mail.username=alert_sa@163.com kafka.eagle.mail.password=mqslimczkdqabbbh kafka.eagle.mail.server.host=smtp.163.com kafka.eagle.mail.server.port=25 ###################################### # alarm im configure ###################################### #kafka.eagle.im.dingding.enable=true #kafka.eagle.im.dingding.url=https://oapi.dingtalk.com/robot/send?access_token= #kafka.eagle.im.wechat.enable=true #kafka.eagle.im.wechat.token=https://qyapi.weixin.qq.com/cgi-bin/gettoken?corpid=xxx&corpsecret=xxx #kafka.eagle.im.wechat.url=https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token= #kafka.eagle.im.wechat.touser= #kafka.eagle.im.wechat.toparty= #kafka.eagle.im.wechat.totag= #kafka.eagle.im.wechat.agentid= ###################################### # delete kafka topic token ###################################### kafka.eagle.topic.token=keadmin ###################################### # kafka sasl authenticate ###################################### cluster1.kafka.eagle.sasl.enable=false cluster1.kafka.eagle.sasl.protocol=SASL_PLAINTEXT cluster1.kafka.eagle.sasl.mechanism=PLAIN cluster1.kafka.eagle.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="kafka-eagle"; # cluster2.kafka.eagle.sasl.enable=false # cluster2.kafka.eagle.sasl.protocol=SASL_PLAINTEXT # cluster2.kafka.eagle.sasl.mechanism=PLAIN # cluster2.kafka.eagle.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="kafka-eagle"; ###################################### # kafka jdbc driver address ###################################### # 默認使用sqlite輕量級數據庫, 可以配置為mysql, oracle, redis, mongodb等數據庫 kafka.eagle.driver=org.sqlite.JDBC kafka.eagle.url=jdbc:sqlite:/opt/ronnie/kafka/kafka-eagle/kafka-eagle-web-1.3.9/db/ke.db kafka.eagle.username=root kafka.eagle.password=123456- 創建JMX端口
vim /opt/ronnie/kafka/bin/kafka-server-start.sh # kafka-server-start.sh 在 export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"后添加JMX端口 if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G" export JMX_PORT="9999" fi- 修改ke.sh權限
cd /opt/ronnie/kafka/kafka-eagle/kafka-eagle-web-1.3.9/bin/ chmod 777 ke.sh- 開啟kafka-eagle
ke.sh start # 創建成功 ******************************************************************* * Kafka Eagle system monitor port successful... ******************************************************************* [2019-10-04 13:27:09] INFO: Status Code[0] [2019-10-04 13:27:09] INFO: [Job done!] Welcome to __ __ ___ ____ __ __ ___ ______ ___ ______ __ ______ / //_/ / | / __/ / //_/ / | / ____/ / | / ____/ / / / ____/ / ,< / /| | / /_ / ,< / /| | / __/ / /| | / / __ / / / __/ / /| | / ___ | / __/ / /| | / ___ | / /___ / ___ |/ /_/ / / /___ / /___ /_/ |_| /_/ |_|/_/ /_/ |_| /_/ |_| /_____/ /_/ |_|\____/ /_____//_____/ Version 1.3.9 ******************************************************************* * Kafka Eagle Service has started success. * Welcome, Now you can visit 'http://192.168.180.200:8048/ke' * Account:admin ,Password:123456 *******************************************************************進入http://192.168.180.200:8048/ke, 以admin為用戶名, 123456為密碼即可進入kafka eagle dashboard
