k8s 部署zk 集群

1. 環境介紹

使用helm 進行部署zookeeper：3.7

2. zk集群部署

helm repo add bitnami	https://charts.bitnami.com/bitnami

3.創建zk集群

#首先使用fetch進行拉取chart
 helm fetch  bitnami/zookeeper
 
#將拉取下zk的壓縮包進行解壓
  tar   xf  zookeeper-6.7.1.tgz
  
#修改values.yaml部分字段
replicaCount: 3  #副本數為3
accessModes:
    - ReadWriteOnce
  size: 10Gi  #根據自己情況設定不能小於10G

serviceMonitor: #開啟prometheus
    enabled: true
    namespace：zk

resources: #對pod進行資源限制
  requests:
    memory: 1Gi
    cpu: "1"
  limits:
    memory: 2Gi
    cpu: "1.5"

helm install   zk  bitnami/zookeeper -f values.yaml  -n zk #安裝zk集群

4.查看pod

[root@baozexu tmp]# kubectl get pods -n zk
NAME   READY   STATUS    RESTARTS   AGE
zk-zookeeper-0    1/1     Running   0          3h36m
zk-zookeeper-1   1/1     Running   0          3h35m
zk-zookeeper-2     Running   0          3h35m

5.查看pvc

[root@baozexu tmp]# kubectl get pvc -n zk
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
zk-zk-0   Bound    pvc-bc74640f-5adb-46e6-ab48-c79f99f7a559   10Gi       RWO            cbs            4h5m
zk-zk-1   Bound    pvc-14f47244-6e7d-480b-8db6-a834ac0b8ea1   10Gi       RWO            cbs            4h4m
zk-zk-2   Bound    zk-zk-2                                    10Gi       RWO            cbs            66m

6.查看zk配置

[root@baozexu zookeeper]# kubectl exec -n zk  zk-zookeeper-0  -- cat /opt/bitnami/zookeeper/conf/zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/bitnami/zookeeper/data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.purgeInterval=0

## Metrics Providers
#
# https://prometheus.io Metrics Exporter
#metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
#metricsProvider.httpPort=7000
#metricsProvider.exportJvmInfo=true
maxCnxns=0
reconfigEnabled=false
quorumListenOnAllIPs=false
4lw.commands.whitelist=srvr, mntr, ruok
maxSessionTimeout=40000
server.1=zk-zookeeper-0.zk-zookeeper-headless.zk.svc.cluster.local:2888:3888;2181
server.2=zk-zookeeper-1.zk-zookeeper-headless.zk.svc.cluster.local:2888:3888;2181
server.3=zk-zookeeper-2.zk-zookeeper-headless.zk.svc.cluster.local:2888:3888;2181

7.查看集群狀態

[root@baozexu tmp]# kubectl exec -n zk  zk-zookeeper-0 zkServer.sh  status
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
ZooKeeper JMX enabled by default
Using config: /usr/bin/../etc/zookeeper/zoo.cfg
Mode: follower
[root@baozexu tmp]# kubectl exec -n zk  zk-zookeeper-1  zkServer.sh  status
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
ZooKeeper JMX enabled by default
Using config: /usr/bin/../etc/zookeeper/zoo.cfg
Mode: leader
[root@baozexu tmp]# kubectl exec -n zk zk-zookeeper-2  zkServer.sh  status
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
ZooKeeper JMX enabled by default
Using config: /usr/bin/../etc/zookeeper/zoo.cfg
Mode: follower

8. zk 日志清理

1. 自動清理

#在zoo.cfg 進行配置
autopurge.snapRetainCount=60 
autopurge.purgeInterval=48
# 指定每隔48小時清理一次zookeeper事務日志,防止事務日志過多占用空間,清理后保留最近60份

2 .手動清理

zkCleanup.sh -n 60 清理后保留最近60份

#直接-n 指定保留的文件數即可,不需要指定目錄,因為配置文件中已經寫了

9.zk 備份

注意該備份工具不支持權限認證

git clone https://github.com/boundary/zoocreeper.git

1. 將項目clone 下來之后進行maven 編譯

mvn clean package

2. 、要查看創建備份的可用選項

$ java -jar target/zoocreeper-1.0-SNAPSHOT.jar --help

3. 要查看還原備份的可用選項，請運行：

$ java -cp target/zoocreeper-1.0-SNAPSHOT.jar \
    com.boundary.zoocreeper.Restore --help

示例：

./zoocreeper dump -z 127.0.0.1 > dumpfile.json # 備份
$ cat dumpfile.json | ./zoocreeper load -z 127.0.0.1 #恢復

由於命令備份太過簡單 寫了shell腳本 進行封裝

備份

#!/bin/bash
Date=$(date +%F-%H-%M)
Dest=/backup
IP=10.7.255.222
Path=/root/zoocreeper
[ -d  $Dest/ ] || mkdir -p $Dest/
sh $Path/zoocreeper dump -z  $IP   > $Dest/zk_backup_${Date}.json

恢復

#!/bin/bash
Dest=/backup
Path=/root/zoocreeper
read -p  "【是否進行恢復[Y/N]】" Action
case  $Action in
Y)
read -p  "【請輸入zk的鏈接地址】" IP
read -p  "【請輸入備份數據的日期 示例：2021-04-23-19-22】" Date
cd $Dest/  \
if [  ! -f zk_backup_${Date}.json ];then
echo "備份文件不存在"
else
cat zk_backup_${Date}.json | sh $Path/zoocreeper load -z $IP
fi
;;
N)
exit 1
esac

9. 監控

# 查看servicemonitor 是否存在
kubectl get servicemonitor -n zk

查看指標是否采集成功

開啟grafana 外網訪問

導入dashboard（https://grafana.com）

10.zk壓測

https://github.com/brownsys/zookeeper-benchmark.git

生成和使用說明

#進行maven 編譯
mvn -DZooKeeperVersion= package

#修改 benchmark.conf 配置文件
totalTime=300000 #壓測時間
totalOperations=200000 #並發連接數
sync=false #同步/異步
server.1=10.7.255.222:2181 #zk的連接地址

#使用配置文件運行壓力測試：
java -cp target/lib/*:target/* edu.brown.cs.zkbenchmark.ZooKeeperBenchmark --conf benchmark.conf

以下截圖來自grafana
測試條件：

環境為k8s部署的3個pod

分別持續5分鍾，操作數為20萬,以下是壓測結果