使用helm在k8s上部署kafka

https://blog.frognew.com/2019/07/use-helm-install-kafka-on-k8s.html

 

1.配置helm chart repo

kafka的helm chart還在孵化當中，使用前需要添加incubator的repo：helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator。

肉身在國內需要設置azure提供的鏡像庫地址:

1 2 3 4 5 6 7 8

helm repo add stable http://mirror.azure.cn/kubernetes/charts helm repo add incubator http://mirror.azure.cn/kubernetes/charts-incubator helm repo list NAME URL stable http://mirror.azure.cn/kubernetes/charts local http://127.0.0.1:8879/charts incubator http://mirror.azure.cn/kubernetes/charts-incubator

2.創建Kafka和Zookeeper的Local PV

2.1 創建Kafka的Local PV

這里的部署環境是本地的測試環境，存儲選擇Local Persistence Volumes。首先，在k8s集群上創建本地存儲的StorageClass local-storage.yaml：

1 2 3 4 5 6 7

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: local-storage provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer reclaimPolicy: Retain

1 2	kubectl apply -f local-storage.yaml storageclass.storage.k8s.io/local-storage created

這里要在node1、node2這兩個k8s節點上部署3個kafka的broker節點，因此先在node1、node2上創建這3個kafka broker節點的Local PV kafka-local-pv.yaml:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

apiVersion: v1 kind: PersistentVolume metadata: name: datadir-kafka-0 spec: capacity: storage: 5Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /home/kafka/data-0 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node1 --- apiVersion: v1 kind: PersistentVolume metadata: name: datadir-kafka-1 spec: capacity: storage: 5Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /home/kafka/data-1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node2 --- apiVersion: v1 kind: PersistentVolume metadata: name: datadir-kafka-2 spec: capacity: storage: 5Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /home/kafka/data-2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node2

1	kubectl apply -f kafka-local-pv.yaml

根據上面創建的local pv，在node1上創建目錄/home/kafka/data-0，在node2上創建目錄/home/kafka/data-1和/home/kafka/data-2。

1 2 3 4 5 6

# node1 mkdir -p /home/kafka/data-0 # node2 mkdir -p /home/kafka/data-1 mkdir -p /home/kafka/data-2

2.2 創建Zookeeper的Local PV

這里要在node1、node2這兩個k8s節點上部署3個zookeeper節點，因此先在node1、node2上創建這3個zookeeper節點的Local PV zookeeper-local-pv.yaml:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

apiVersion: v1 kind: PersistentVolume metadata: name: data-kafka-zookeeper-0 spec: capacity: storage: 5Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /home/kafka/zkdata-0 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node1 --- apiVersion: v1 kind: PersistentVolume metadata: name: data-kafka-zookeeper-1 spec: capacity: storage: 5Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /home/kafka/zkdata-1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node2 --- apiVersion: v1 kind: PersistentVolume metadata: name: data-kafka-zookeeper-2 spec: capacity: storage: 5Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /home/kafka/zkdata-2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node2

1	kubectl apply -f zookeeper-local-pv.yaml

根據上面創建的local pv，在node1上創建目錄/home/kafka/zkdata-0，在node2上創建目錄/home/kafka/zkdata-1和/home/kafka/zkdata-2。

1 2 3 4 5 6

# node1 mkdir -p /home/kafka/zkdata-0 # node2 mkdir -p /home/kafka/zkdata-1 mkdir -p /home/kafka/zkdata-2

3.部署Kafka

編寫kafka chart的vaule文件kafka-values.yaml:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

replicas: 3 tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - key: node-role.kubernetes.io/master operator: Exists effect: PreferNoSchedule persistence: storageClass: local-storage size: 5Gi zookeeper: persistence: enabled: true storageClass: local-storage size: 5Gi replicaCount: 3 image: repository: gcr.azk8s.cn/google_samples/k8szk tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - key: node-role.kubernetes.io/master operator: Exists effect: PreferNoSchedule

• 安裝過程需要使用到gcr.io/google_samples/k8szk:v3等docker鏡像，切換成使用azure的GCR Proxy Cache：gcr.azk8s.cn。

  1	helm install --name kafka --namespace kafka -f kafka-values.yaml incubator/kafka

最后需要確認所有的pod都處於running狀態:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

kubectl get pod -n kafka -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kafka-0 1/1 Running 0 12m 10.244.0.61 node1 <none> <none> kafka-1 1/1 Running 0 6m3s 10.244.1.12 node2 <none> <none> kafka-2 1/1 Running 0 2m26s 10.244.1.13 node2 <none> <none> kafka-zookeeper-0 1/1 Running 0 12m 10.244.1.9 node2 <none> <none> kafka-zookeeper-1 1/1 Running 0 11m 10.244.1.10 node2 <none> <none> kafka-zookeeper-2 1/1 Running 0 11m 10.244.1.11 node2 <none> <none> kubectl get statefulset -n kafka NAME READY AGE kafka 3/3 22m kafka-zookeeper 3/3 22m kubectl get service -n kafka NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kafka ClusterIP 10.102.8.192 <none> 9092/TCP 31m kafka-headless ClusterIP None <none> 9092/TCP 31m kafka-zookeeper ClusterIP 10.110.43.203 <none> 2181/TCP 31m kafka-zookeeper-headless ClusterIP None <none> 2181/TCP,3888/TCP,2888/TCP 31m

可以看到當前kafka的helm chart，采用StatefulSet的形式部署了kafka和zookeeper，而我們通過Local PV的形式，將kafka-0調度到node1上，將kafka-1和kafka-2調度到node2上。

4.安裝后的測試

在k8s集群內運行下面的客戶端Pod，訪問kafka broker進行測試：

1 2 3 4 5 6 7 8 9 10 11 12 13

apiVersion: v1 kind: Pod metadata: name: testclient namespace: kafka spec: containers: - name: kafka image: confluentinc/cp-kafka:5.0.1 command: - sh - -c - "exec tail -f /dev/null"

創建並進入testclient容器內：

1 2	kubectl apply -f testclient.yaml kubectl -n kafka exec testclient -it sh

查看kafka相關命令：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

ls /usr/bin/ | grep kafka kafka-acls kafka-broker-api-versions kafka-configs kafka-console-consumer kafka-console-producer kafka-consumer-groups kafka-consumer-perf-test kafka-delegation-tokens kafka-delete-records kafka-dump-log kafka-log-dirs kafka-mirror-maker kafka-preferred-replica-election kafka-producer-perf-test kafka-reassign-partitions kafka-replica-verification kafka-run-class kafka-server-start kafka-server-stop kafka-streams-application-reset kafka-topics kafka-verifiable-consumer kafka-verifiable-producer

創建一個Topic test1：

1	kafka-topics --zookeeper kafka-zookeeper:2181 --topic test1 --create --partitions 1 --replication-factor 1

查看的Topic：

1 2	kafka-topics --zookeeper kafka-zookeeper:2181 --list test1

5.總結

當前基於Helm官方倉庫的chartincubator/kafka在k8s上部署的kafka，使用的鏡像是confluentinc/cp-kafka:5.0.1。 即部署的是Confluent公司提供的kafka版本。Confluent Platform Kafka(簡稱CP Kafka)提供了一些Apache Kafka沒有的高級特性，例如跨數據中心備份、Schema注冊中心以及集群監控工具等。CP Kafka目前分為免費版本和企業版兩種，免費版除了Apache Kafka的標准組件外還包含Schema注冊中心和Rest Proxy。

Confluent Platform and Apache Kafka Compatibility中給出了Confluent Kafka和Apache Kafka的版本對應關系，可以看出這里安裝的cp 5.0.1對應Apache Kafka的2.0.x。

進入一個broker容器中，查看：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

ls /usr/share/java/kafka | grep kafka kafka-clients-2.0.1-cp1.jar kafka-log4j-appender-2.0.1-cp1.jar kafka-streams-2.0.1-cp1.jar kafka-streams-examples-2.0.1-cp1.jar kafka-streams-scala_2.11-2.0.1-cp1.jar kafka-streams-test-utils-2.0.1-cp1.jar kafka-tools-2.0.1-cp1.jar kafka.jar kafka_2.11-2.0.1-cp1-javadoc.jar kafka_2.11-2.0.1-cp1-scaladoc.jar kafka_2.11-2.0.1-cp1-sources.jar kafka_2.11-2.0.1-cp1-test-sources.jar kafka_2.11-2.0.1-cp1-test.jar kafka_2.11-2.0.1-cp1.jar

可以看到對應apache kafka的版本號是2.11-2.0.1，前面2.11是Scala編譯器的版本，Kafka的服務器端代碼是使用Scala語言開發的，后邊2.0.1是Kafka的版本。 即CP Kafka 5.0.1是基於Apache Kafka 2.0.1的。