Kubernetes K8S之Helm部署ELK日志分析系統；由於Logstash比較消耗資源，因此本次我們使用Fluentd實現日志收集（EFK）。
主機配置規划
| 服務器名稱(hostname) | 系統版本 | 配置 | 內網IP | 外網IP(模擬) |
|---|---|---|---|---|
| k8s-master | CentOS7.7 | 2C/4G/20G | 172.16.1.110 | 10.0.0.110 |
| k8s-node01 | CentOS7.7 | 2C/4G/20G | 172.16.1.111 | 10.0.0.111 |
| k8s-node02 | CentOS7.7 | 2C/4G/20G | 172.16.1.112 | 10.0.0.112 |
備注：由於EFK部署比較消耗內存；所以每台機器的內存最好大於等於4G。
ELK概述
ELK是Elasticsearch、Logstash、Kibana的簡稱，這三者是核心套件，但並非全部。
Elasticsearch是實時全文搜索和分析引擎，提供搜集、分析、存儲數據三大功能；是一套開放REST和JAVA API等接口，提供高效搜索功能，可擴展的分布式系統。它構建於Apache Lucene搜索引擎庫之上。
Logstash是一個用來搜集、分析、過濾日志的工具。它支持幾乎任何類型的日志，包括系統日志、錯誤日志和自定義應用程序日志。它可以從許多來源接收日志，這些來源包括 syslog、消息傳遞（例如 RabbitMQ）和JMX，它能夠以多種方式輸出數據，包括電子郵件、websockets和Elasticsearch。
Kibana是一個基於Web的圖形界面，用於搜索、分析和可視化存儲在 Elasticsearch指標中的日志數據。它利用Elasticsearch的REST接口來檢索數據，不僅允許用戶創建他們自己數據定制儀表板的視圖，還允許他們以特殊的方式查詢和過濾數據。
由於Logstash比較消耗資源，因此本次我們使用Fluentd實現日志收集（EFK）。
EFK鏡像下載
由於鏡像都在國外，因此我們在國內下載鏡像，然后tag為對應的鏡像名稱。執行如下腳本【集群所有機器都執行】：
1 [root@k8s-node02 software]# vim download_efk_image.sh 2 #!/bin/sh 3 4 ##### 在 master 節點和 worker 節點都要執行 【所有機器執行】 5 6 # 加載環境變量 7 . /etc/profile 8 . /etc/bashrc 9 10 # 變量設置 11 elasticsearch_iamge="elasticsearch-oss:6.7.0" 12 busybox_image="busybox:latest" 13 bats_image="bats:0.4.0" 14 fluentd_image="fluentd-elasticsearch:v2.3.2" 15 kibana_image="kibana-oss:6.7.0" 16 17 # 集群所有機器執行 18 # elasticsearch鏡像下載 19 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${elasticsearch_iamge} 20 docker tag registry.cn-beijing.aliyuncs.com/google_registry/${elasticsearch_iamge} docker.elastic.co/elasticsearch/${elasticsearch_iamge} 21 docker rmi registry.cn-beijing.aliyuncs.com/google_registry/${elasticsearch_iamge} 22 # busybox鏡像下載 23 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${busybox_image} 24 docker tag registry.cn-beijing.aliyuncs.com/google_registry/${busybox_image} ${busybox_image} 25 docker rmi registry.cn-beijing.aliyuncs.com/google_registry/${busybox_image} 26 # bats鏡像下載 27 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${bats_image} 28 docker tag registry.cn-beijing.aliyuncs.com/google_registry/${bats_image} dduportal/${bats_image} 29 docker rmi registry.cn-beijing.aliyuncs.com/google_registry/${bats_image} 30 # fluentd-elasticsearch鏡像下載 31 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${fluentd_image} 32 docker tag registry.cn-beijing.aliyuncs.com/google_registry/${fluentd_image} gcr.io/google-containers/${fluentd_image} 33 docker rmi registry.cn-beijing.aliyuncs.com/google_registry/${fluentd_image} 34 # kibana-oss鏡像下載 35 docker pull registry.cn-beijing.aliyuncs.com/google_registry/${kibana_image} 36 docker tag registry.cn-beijing.aliyuncs.com/google_registry/${kibana_image} docker.elastic.co/kibana/${kibana_image} 37 docker rmi registry.cn-beijing.aliyuncs.com/google_registry/${kibana_image}
Elasticsearch部署
本次部署EFK，創建一個efk名稱空間。
chart下載與配置修改
1 # 當前目錄 2 [root@k8s-master efk]# pwd 3 /root/k8s_practice/efk 4 # 創建 efk 名稱空間 5 [root@k8s-master efk]# kubectl create namespace efk 6 [root@k8s-master efk]# 7 # ES版本查看，本次我們部署chart 1.30.0版本，ES 6.7.0版本 8 [root@k8s-master efk]# helm search stable/elasticsearch -l 9 NAME CHART VERSION APP VERSION DESCRIPTION 10 stable/elasticsearch 1.32.5 6.8.6 DEPRECATED Flexible and powerful open source, distributed... 11 stable/elasticsearch 1.32.4 6.8.6 Flexible and powerful open source, distributed real-time ... 12 stable/elasticsearch 1.32.3 6.8.6 Flexible and powerful open source, distributed real-time ... 13 ……………… 14 [root@k8s-master efk]# 15 [root@k8s-master efk]# helm fetch stable/elasticsearch --version 1.30.0 16 [root@k8s-master efk]# tar xf elasticsearch-1.30.0.tgz 17 # 修改配置文件1 18 [root@k8s-master efk]# vim elasticsearch/values.yaml 19 initImage: 20 repository: "busybox" 21 tag: "latest" 22 pullPolicy: "IfNotPresent" # 從Always 改為IfNotPresent 23 ……………… 24 client: 25 name: client 26 replicas: 1 # 從2改為1，因為是在自己PC機操作的，內存有限 27 serviceType: ClusterIP 28 ……………… 29 master: 30 name: master 31 exposeHttp: false 32 replicas: 3 # 不要修改 33 heapSize: "512m" 34 persistence: 35 enabled: false # 沒有多余的PVC，因此從true改為false 36 accessMode: ReadWriteOnce 37 ……………… 38 data: 39 name: data 40 exposeHttp: false 41 replicas: 1 # 從2改為1，因為是在自己PC機操作的，內存有限 42 heapSize: "1024m" # 從1536m改為1024m，因為是在自己PC機操作的，內存有限 43 persistence: 44 enabled: false # 沒有多余的PVC，因此從true改為false 45 accessMode: ReadWriteOnce 46 [root@k8s-master efk]# 47 # 修改配置文件2 48 [root@k8s-master efk]# vim elasticsearch/templates/client-deployment.yaml 49 apiVersion: apps/v1 # 從 apps/v1beta1 改為 apps/v1 50 kind: Deployment 51 ……………… 52 spec: 53 replicas: {{ .Values.client.replicas }} 54 # 添加信息 如下 55 selector: 56 matchLabels: 57 app: {{ template "elasticsearch.name" . }} 58 component: "{{ .Values.client.name }}" 59 release: {{ .Release.Name }} 60 # 添加信息 如上 61 [root@k8s-master efk]# 62 # 修改配置文件3 63 [root@k8s-master efk]# vim elasticsearch/templates/data-statefulset.yaml 64 apiVersion: apps/v1 # 從 apps/v1beta1 改為 apps/v1 65 kind: StatefulSet 66 ……………… 67 spec: 68 serviceName: {{ template "elasticsearch.data.fullname" . }} 69 replicas: {{ .Values.data.replicas }} 70 # 添加信息 如下 71 selector: 72 matchLabels: 73 app: {{ template "elasticsearch.name" . }} 74 component: "{{ .Values.data.name }}" 75 release: {{ .Release.Name }} 76 role: data 77 # 添加信息 如上 78 [root@k8s-master efk]# 79 # 修改配置文件4 80 [root@k8s-master efk]# vim elasticsearch/templates/master-statefulset.yaml 81 apiVersion: apps/v1 # 從 apps/v1beta1 改為 apps/v1 82 kind: StatefulSet 83 ……………… 84 spec: 85 serviceName: {{ template "elasticsearch.master.fullname" . }} 86 replicas: {{ .Values.master.replicas }} 87 # 添加信息 如下 88 selector: 89 matchLabels: 90 app: {{ template "elasticsearch.name" . }} 91 component: "{{ .Values.master.name }}" 92 release: {{ .Release.Name }} 93 role: master 94 # 添加信息 如上 95 [root@k8s-master efk]#
Elasticsearch部署
步驟如下：
1 # 當前目錄 2 [root@k8s-master efk]# pwd 3 /root/k8s_practice/efk 4 # 部署ES 5 [root@k8s-master efk]# helm install --name es01 --namespace=efk elasticsearch/ 6 [root@k8s-master efk]# 7 # 狀態查看 8 [root@k8s-master ~]# helm list 9 NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE 10 es01 1 Sat Jul 25 12:18:50 2020 DEPLOYED elasticsearch-1.30.0 6.7.0 efk 11 [root@k8s-master efk]# 12 # 等待一會兒后【估計幾分鍾】，查看pod狀態信息如下 13 [root@k8s-master ~]# kubectl get deploy -n efk 14 NAME READY UP-TO-DATE AVAILABLE AGE 15 es01-elasticsearch-client 1/1 1 1 6m13s 16 [root@k8s-master ~]# 17 [root@k8s-master ~]# kubectl get pod -n efk 18 NAME READY STATUS RESTARTS AGE 19 es01-elasticsearch-client-646f8f866d-rt2wp 1/1 Running 0 6m21s 20 es01-elasticsearch-data-0 1/1 Running 0 6m21s 21 es01-elasticsearch-master-0 1/1 Running 0 6m21s 22 es01-elasticsearch-master-1 1/1 Running 0 5m30s 23 es01-elasticsearch-master-2 1/1 Running 0 5m3s 24 [root@k8s-master efk]# 25 [root@k8s-master efk]# kubectl get svc -n efk 26 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE 27 es01-elasticsearch-client ClusterIP 10.100.237.152 <none> 9200/TCP 6m34s 28 es01-elasticsearch-discovery ClusterIP None <none> 9300/TCP 6m42s 29 [root@k8s-master efk]# 30 [root@k8s-master efk]# kubectl get sts -n efk 31 NAME READY AGE 32 es01-elasticsearch-data 1/1 7m4s 33 es01-elasticsearch-master 3/3 7m4s 34 [root@k8s-master efk]#
Elasticsearch訪問
其中IP來源於ES的svc。
1 [root@k8s-master ~]# curl 10.100.237.152:9200/ 2 { 3 "name" : "es01-elasticsearch-client-646f8f866d-rt2wp", 4 "cluster_name" : "elasticsearch", 5 "cluster_uuid" : "S4t_UDOuRye9mtK22VWxLw", 6 "version" : { 7 "number" : "6.7.0", 8 "build_flavor" : "oss", 9 "build_type" : "docker", 10 "build_hash" : "8453f77", 11 "build_date" : "2019-03-21T15:32:29.844721Z", 12 "build_snapshot" : false, 13 "lucene_version" : "7.7.0", 14 "minimum_wire_compatibility_version" : "5.6.0", 15 "minimum_index_compatibility_version" : "5.0.0" 16 }, 17 "tagline" : "You Know, for Search" 18 } 19 [root@k8s-master ~]# 20 [root@k8s-master ~]# curl 10.100.237.152:9200/_cluster/health?pretty 21 { 22 "cluster_name" : "elasticsearch", 23 "status" : "green", # 可見狀態正常 24 "timed_out" : false, 25 "number_of_nodes" : 5, 26 "number_of_data_nodes" : 1, 27 "active_primary_shards" : 0, 28 "active_shards" : 0, 29 "relocating_shards" : 0, 30 "initializing_shards" : 0, 31 "unassigned_shards" : 0, 32 "delayed_unassigned_shards" : 0, 33 "number_of_pending_tasks" : 0, 34 "number_of_in_flight_fetch" : 0, 35 "task_max_waiting_in_queue_millis" : 0, 36 "active_shards_percent_as_number" : 100.0 37 }
至此，elasticsearch部署完畢
elasticsearch-client域名獲取
根據es01-elasticsearch-client的svc信息獲取到es01-elasticsearch-client的域名；用於后面的fluentd 和kibana。
啟動一個pod
1 [root@k8s-master test]# pwd 2 /root/k8s_practice/test 3 [root@k8s-master test]# cat myapp_demo.yaml 4 apiVersion: v1 5 kind: Pod 6 metadata: 7 name: myapp-demo 8 namespace: default 9 labels: 10 k8s-app: myapp 11 spec: 12 containers: 13 - name: myapp 14 image: registry.cn-beijing.aliyuncs.com/google_registry/myapp:v1 15 imagePullPolicy: IfNotPresent 16 ports: 17 - name: httpd 18 containerPort: 80 19 protocol: TCP 20 [root@k8s-master test]# 21 [root@k8s-master test]# kubectl apply -f myapp_demo.yaml 22 pod/myapp-demo created 23 [root@k8s-master test]# 24 [root@k8s-master test]# kubectl get pod -o wide 25 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 26 myapp-demo 1/1 Running 0 6s 10.244.2.84 k8s-node02 <none> <none>
進入pod並得到elasticsearch-client域名信息
1 # 進入一個pod容器 2 [root@k8s-master test]# kubectl exec -it myapp-demo sh 3 ##### 格式 nslookup svc-cluster-ip 4 / # nslookup 10.100.237.152 5 nslookup: can't resolve '(null)': Name does not resolve 6 7 Name: 10.100.237.152 8 Address 1: 10.100.237.152 es01-elasticsearch-client.efk.svc.cluster.local 9 / # 10 / # 11 ### 通過ping判斷域名是否通暢 12 / # ping es01-elasticsearch-client.efk.svc.cluster.local 13 PING es01-elasticsearch-client.efk.svc.cluster.local (10.100.237.152): 56 data bytes 14 64 bytes from 10.100.237.152: seq=0 ttl=64 time=0.094 ms 15 64 bytes from 10.100.237.152: seq=1 ttl=64 time=0.081 ms 16 64 bytes from 10.100.237.152: seq=2 ttl=64 time=0.243 ms
由上可得，Service中es01-elasticsearch-client的域名為：es01-elasticsearch-client.efk.svc.cluster.local
Service的域名格式為：$(service name).$(namespace).svc.cluster.local，其中 cluster.local 指定的集群的域名
Fluentd部署
chart下載與配置修改
1 [root@k8s-master efk]# pwd 2 /root/k8s_practice/efk 3 [root@k8s-master efk]# 4 # fluentd版本信息查看 5 [root@k8s-master efk]# helm search stable/fluentd-elasticsearch -l 6 NAME CHART VERSION APP VERSION DESCRIPTION 7 stable/fluentd-elasticsearch 2.0.7 2.3.2 DEPRECATED! - A Fluentd Helm chart for Kubernetes with El... 8 stable/fluentd-elasticsearch 2.0.6 2.3.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou... 9 stable/fluentd-elasticsearch 2.0.5 2.3.2 A Fluentd Helm chart for Kubernetes with Elasticsearch ou... 10 ……………… 11 # 獲取fluentd-elasticsearch 並解壓 12 [root@k8s-master efk]# helm fetch stable/fluentd-elasticsearch --version 2.0.7 13 [root@k8s-master efk]# tar xf fluentd-elasticsearch-2.0.7.tgz 14 # 配置修改 15 [root@k8s-master efk]# vim fluentd-elasticsearch/values.yaml 16 ### 為什么使用域名而不是IP，因此每次重啟ES的svc，對應IP都會改變。而域名是不變的 17 elasticsearch: 18 host: 'es01-elasticsearch-client.efk.svc.cluster.local' # 修改處，域名獲取參見上文 19 port: 9200 20 scheme: 'http' 21 [root@k8s-master efk]#
fluentd-elasticsearch部署
步驟如下：
1 ################ 部署fluentd-elasticsearch 2 # 當前目錄 3 [root@k8s-master efk]# pwd 4 /root/k8s_practice/efk 5 # 部署fluentd-elasticsearch 6 [root@k8s-master efk]# helm install --name fluentd-es01 --namespace=efk fluentd-elasticsearch 7 [root@k8s-master efk]# 8 # 狀態查看 9 [root@k8s-master efk]# helm list 10 NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE 11 es01 1 Sat Jul 25 12:18:50 2020 DEPLOYED elasticsearch-1.30.0 6.7.0 efk 12 fluentd-es01 1 Sat Jul 25 12:36:01 2020 DEPLOYED fluentd-elasticsearch-2.0.7 2.3.2 efk 13 [root@k8s-master efk]# 14 #查看pod狀態信息如下 15 [root@k8s-master efk]# kubectl get ds -n efk 16 NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE 17 fluentd-es01-fluentd-elasticsearch 2 2 2 2 2 <none> 113s 18 [root@k8s-master efk]# 19 [root@k8s-master efk]# kubectl get pod -n efk -o wide 20 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 21 es01-elasticsearch-client-646f8f866d-rt2wp 1/1 Running 0 17m 10.244.2.57 k8s-node02 <none> <none> 22 es01-elasticsearch-data-0 1/1 Running 0 17m 10.244.2.58 k8s-node02 <none> <none> 23 es01-elasticsearch-master-0 1/1 Running 0 17m 10.244.4.241 k8s-node01 <none> <none> 24 es01-elasticsearch-master-1 1/1 Running 0 17m 10.244.2.59 k8s-node02 <none> <none> 25 es01-elasticsearch-master-2 1/1 Running 0 16m 10.244.4.242 k8s-node01 <none> <none> 26 fluentd-es01-fluentd-elasticsearch-qnmf9 1/1 Running 0 43s 10.244.4.243 k8s-node01 <none> <none> 27 fluentd-es01-fluentd-elasticsearch-xmw5f 1/1 Running 0 43s 10.244.2.60 k8s-node02 <none> <none>
Kibana部署
kibana的主版本和大版本必須和elasticsearch（ES）一致，小版本可以不同；但兩者版本最好一致，這樣可以避免一些因版本不同導致的意外情況。
由於elasticsearch（ES）使用的是6.7.0，因此kibana我們也是用該版本。
chart下載與配置修改
1 [root@k8s-master efk]# pwd 2 /root/k8s_practice/efk 3 # 所有版本查看 4 [root@k8s-master efk]# helm search stable/kibana -l 5 NAME CHART VERSION APP VERSION DESCRIPTION 6 stable/kibana 3.2.7 6.7.0 Kibana is an open source data visualization plugin for El... 7 stable/kibana 3.2.6 6.7.0 Kibana is an open source data visualization plugin for El... 8 ……………… 9 # 獲取kibana，並解壓縮 10 [root@k8s-master efk]# helm fetch stable/kibana --version 3.2.7 11 [root@k8s-master efk]# tar xf kibana-3.2.7.tgz 12 # 配置修改1 13 [root@k8s-master efk]# vim kibana/values.yaml 14 ### 為什么使用域名而不是IP，因此每次重啟ES的svc，對應IP都會改變。而域名是不變的 15 files: 16 kibana.yml: 17 ## Default Kibana configuration from kibana-docker. 18 server.name: kibana 19 server.host: "0" 20 elasticsearch.url: http://es01-elasticsearch-client.efk.svc.cluster.local:9200 # 修改處，域名獲取參見上文 21 ……………… 22 service: 23 type: NodePort # 修改內容 從ClusterIP改為NodePort 24 externalPort: 443 25 internalPort: 5601 26 nodePort: 30601 # 添加處，Service端口范圍：30000-32767 27 [root@k8s-master efk]# 28 # 配置修改2 29 [root@k8s-master efk]# vim kibana/templates/deployment.yaml 30 apiVersion: apps/v1 # 從 apps/v1beta1 改為 apps/v1 31 kind: Deployment 32 metadata: 33 ……………… 34 spec: 35 replicas: {{ .Values.replicaCount }} 36 revisionHistoryLimit: {{ .Values.revisionHistoryLimit }} 37 # 添加信息 如下 38 selector: 39 matchLabels: 40 app: {{ template "kibana.name" . }} 41 release: "{{ .Release.Name }}" 42 # 添加信息 如上
kibana部署
步驟如下：
1 ################ 部署kibana-oss 2 # 當前目錄 3 [root@k8s-master efk]# pwd 4 /root/k8s_practice/efk 5 # 部署kibana-oss 6 [root@k8s-master efk]# helm install --name kibana01 --namespace=efk kibana 7 [root@k8s-master efk]# 8 # 狀態查看 9 [root@k8s-master efk]# helm list 10 NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE 11 es01 1 Sat Jul 25 12:18:50 2020 DEPLOYED elasticsearch-1.30.0 6.7.0 efk 12 fluentd-es01 1 Sat Jul 25 12:36:01 2020 DEPLOYED fluentd-elasticsearch-2.0.7 2.3.2 efk 13 kibana01 1 Sat Jul 25 12:38:18 2020 DEPLOYED kibana-3.2.7 6.7.0 efk 14 [root@k8s-master efk]# 15 #查看pod狀態信息如下 16 [root@k8s-master efk]# kubectl get deploy -n efk 17 NAME READY UP-TO-DATE AVAILABLE AGE 18 es01-elasticsearch-client 1/1 1 1 19m 19 kibana01 1/1 1 1 27s 20 [root@k8s-master efk]# 21 [root@k8s-master efk]# kubectl get pod -n efk -o wide 22 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 23 es01-elasticsearch-client-646f8f866d-rt2wp 1/1 Running 0 20m 10.244.2.57 k8s-node02 <none> <none> 24 es01-elasticsearch-data-0 1/1 Running 0 20m 10.244.2.58 k8s-node02 <none> <none> 25 es01-elasticsearch-master-0 1/1 Running 0 20m 10.244.4.241 k8s-node01 <none> <none> 26 es01-elasticsearch-master-1 1/1 Running 0 19m 10.244.2.59 k8s-node02 <none> <none> 27 es01-elasticsearch-master-2 1/1 Running 0 19m 10.244.4.242 k8s-node01 <none> <none> 28 fluentd-es01-fluentd-elasticsearch-qnmf9 1/1 Running 0 3m10s 10.244.4.243 k8s-node01 <none> <none> 29 fluentd-es01-fluentd-elasticsearch-xmw5f 1/1 Running 0 3m10s 10.244.2.60 k8s-node02 <none> <none> 30 kibana01-bc479f8c7-kr2n2 1/1 Running 0 53s 10.244.4.244 k8s-node01 <none> <none> 31 [root@k8s-master efk]# 32 # 查看svc信息 33 [root@k8s-master efk]# kubectl get svc -n efk -o wide 34 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 35 es01-elasticsearch-client ClusterIP 10.100.237.152 <none> 9200/TCP 20m app=elasticsearch,component=client,release=es01 36 es01-elasticsearch-discovery ClusterIP None <none> 9300/TCP 20m app=elasticsearch,component=master,release=es01 37 kibana01 NodePort 10.101.200.177 <none> 443:30601/TCP 71s app=kibana,release=kibana01
瀏覽器訪問
http://172.16.1.110:30601/
頁面訪問
創建索引
數據查看
相關閱讀
完畢！
———END———
如果覺得不錯就關注下唄 (-^O^-) ！
