基於k8s搭建微服務日志收集中心

由於項目組業務需求，在k8s中運行的微服務日志要進行收集。

1、需要將k8s中的每個pod微服務掛載一個卷，（每個pod都會在一個node上，在node節點上創建一個文件夾，如：/logs/ 將這個目錄掛載到每個微服務的pod中，使之實現映射）

apiVersion: apps/v1
kind: Deployment
metadata:
  name: neusipo-cloud-gateway-deployment
  namespace: gateway
spec:
  selector:
    matchLabels:
      app: neusipo-cloud-gateway
  replicas: 3
  minReadySeconds: 10
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      name: neusipo-cloud-gateway
      labels:
        app: neusipo-cloud-gateway
    spec:
      containers:
      - name: neusipo-cloud-gateway
        image: cr.registry.cnipaig1.cloud/neusipo/neusipo-cloud-gateway-test:1.0  #這里的鏡像地址是阿里雲鏡像倉庫地址
        args: ["--spring.profiles.active=test"]   #配置文件中的測試配置
        env:
        - name: SERVER_PORT
          value: "18080"
        resources:
          requests:
            cpu: 4
            memory: 4Gi
          limits:
            cpu: 4
            memory: 4Gi
        volumeMounts:   #pod中掛載
        - name: gateway-logs   #name與下方本機name保持一致
          mountPath: /sipo-data/neusipo-test/   #pod中的掛載地址
          readOnly: false
      volumes:    #本機卷
      - name: gateway-logs     
        hostPath:
          path: /logs/   #本機創建的掛載目錄 

2、執行命令，創建pod。

kubectl apply -f deployment-demo.yaml -n gateway

3、查看是否創建成功

[root@iZqj001np4kv147ax0tr54Z ~]# kubectl get pods -n gateway
NAME                                                READY   STATUS    RESTARTS   AGE
neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht   1/1     Running   0          4h16m
neusipo-cloud-gateway-deployment-66b6c96fbf-r7wcd   1/1     Running   0          4h16m
neusipo-cloud-gateway-deployment-66b6c96fbf-wwjpk   1/1     Running   0          4h16m

 看到STATUS狀態為Running 時說明容器已啟動。

4、此時驗證pod中是否已經掛載上本機目錄。

[root@iZqj001np4kv147ax0tr54Z ~]# kubectl exec -it  neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht -n gateway -- bash
[root@neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht data]# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          40G   17G   21G  45% /
tmpfs            64M     0   64M   0% /dev
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/vda1        40G   17G   21G  45% /sipo-data/neusipo-test   #這里就代表已經掛載上本機所創建的目錄了
shm              64M     0   64M   0% /dev/shm
tmpfs           7.8G   12K  7.8G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           7.8G     0  7.8G   0% /proc/acpi
tmpfs           7.8G     0  7.8G   0% /proc/scsi
tmpfs           7.8G     0  7.8G   0% /sys/firmware
[root@neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht data]# 

此時查看pod服務啟動后是否有日志打印，通過映射到node節點上創建的目錄。（查看pod在那個node節點上，登錄node節點，進入創建的目錄，查看是否pod日志映射出來）

[root@neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht data]# cd /sipo-data/neusipo-test/
[root@neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht neusipo-test]# ls
gateway
[root@neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht neusipo-test]# cd gateway/
[root@neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht gateway]# ls
logs
[root@neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht gateway]# cd logs/
[root@neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht logs]# ls
error  info　

[root@iZqj001np4kv147ax0tr54Z ~]# kubectl  get pods -n gateway -o wide
NAME                                                READY   STATUS    RESTARTS   AGE     IP             NODE                                         NOMINATED NODE   READINESS GATES
neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht   1/1     Running   0          4h26m   172.20.1.91    cn-beijing-gzj1-d01.i-qj001np4kv147gu45x2l   <none>           <none>
neusipo-cloud-gateway-deployment-66b6c96fbf-r7wcd   1/1     Running   0          4h26m   172.20.0.198   cn-beijing-gzj1-d01.i-qj001np4kv147gu45x2k   <none>           <none>
neusipo-cloud-gateway-deployment-66b6c96fbf-wwjpk   1/1     Running   0          4h26m   172.20.1.60    cn-beijing-gzj1-d01.i-qj001np4kv147gu45x2j   <none>           <none>
[root@iZqj001np4kv147ax0tr54Z ~]# kubectl  get node -o wide
NAME                                         STATUS   ROLES    AGE    VERSION            INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
cn-beijing-gzj1-d01.i-qj001np4kv147ax0tr54   Ready    master   198d   v1.14.8-aliyun.1   10.160.22.68   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.2
cn-beijing-gzj1-d01.i-qj001np4kv147ax0tr55   Ready    master   198d   v1.14.8-aliyun.1   10.160.22.69   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.2
cn-beijing-gzj1-d01.i-qj001np4kv147ax0tr56   Ready    master   198d   v1.14.8-aliyun.1   10.160.22.70   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.2
cn-beijing-gzj1-d01.i-qj001np4kv147gu45x2j   Ready    <none>   198d   v1.14.8-aliyun.1   10.160.22.73   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.2
cn-beijing-gzj1-d01.i-qj001np4kv147gu45x2k   Ready    <none>   198d   v1.14.8-aliyun.1   10.160.22.72   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.2
cn-beijing-gzj1-d01.i-qj001np4kv147gu45x2l   Ready    <none>   198d   v1.14.8-aliyun.1   10.160.22.71   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.2

 從這里看到其中一個pod在10.160.22.73這個node節點上，登錄node節點查看是否有映射的日志

[root@iZqj001np4kv147gu45x2jZ logs]# pwd
/logs
[root@iZqj001np4kv147gu45x2jZ logs]# ls
gateway
[root@iZqj001np4kv147gu45x2jZ logs]# cd gateway/
[root@iZqj001np4kv147gu45x2jZ gateway]# ls
logs
[root@iZqj001np4kv147gu45x2jZ gateway]# cd logs/
[root@iZqj001np4kv147gu45x2jZ logs]# ls
error  info

 如果node節點出現了和pod中一樣的目錄則說明，掛載成功並映射成功，（其他微服務同理就行）

5、日志映射出來后，需要搭建filebeat鏡像，創建filebeat的pod容器，容器需要創建為DaemonSet形式的（DaemonSet所創建的容器類似於守護進程，它會在每個node節點都創建一個pod），使filebeat容器可以從所有的node節點日志目錄拉取日志到pod中，並發送給elk進行日志分析。在master節點上下載filebeat壓縮包，並解壓。

[root@iZqj001np4kv147ax0tr54Z ~]# ls
deployment-demo.yaml              filebeat-7.13.1-linux-x86_64.tar.gz         daemonset-demo.yamlfilebeat-7.13.1-linux-x86_64      daemonset-demo.yaml        dockerfile-filebeat

 創建filebeat的docker鏡像

[root@iZqj001np4kv147ax0tr54Z ~]# cat dockerfile-filebeat 
FROM cr.registry.cnipaig1.cloud/neusipo/centos:7.9.2009
ENV LANG en_US.UTF-8
ENV TZ Asia/Shanghai

ADD filebeat-7.13.1-linux-x86_64.tar.gz /data/

WORKDIR /data/filebeat-7.13.1-linux-x86_64

ENTRYPOINT ["./filebeat", "-c", "filebeat.yml", "-e"]

 上傳到阿里雲的鏡像倉庫，阿里雲容器鏡像服務文檔中已寫如何上傳，照着一步步執行就可。

 

 

 構建filebeat的yaml文件

apiVersion: apps/v1
kind: DaemonSet   #這里為DaemonSet模式
metadata:
  name: neusipo-filebeat
  namespace: filebeat
spec:
  selector:
    matchLabels:
      app: neusipo-filebeat
  template:
    metadata:
      name: neusipo-filebeat
      labels:
        app: neusipo-filebeat
    spec:
      containers:
      - name: neusipo-filebeat
        image: cr.registry.cnipaig1.cloud/neusipo/filebeat:v2
        command: ["./filebeat", "-c", "/data/filebeat/conf/filebeat.yml", "-e"]   #command模塊可以強制替換容器里默認的啟動命令
        resources:
          requests:
            cpu: 4
            memory: 4Gi
          limits:
            cpu: 4
            memory: 4Gi
        volumeMounts:
          - name: filebeat-log
            mountPath: /sipo-data/neusipo-test/    #這里為掛載node目錄到pod中的目錄
            readOnly: false
          - name: neusipo-filebeat
            mountPath: /data/filebeat/conf/       #這里為pod容器內filebeat配置文件目錄
            readOnly: true
      volumes:
      - name: filebeat-log
        hostPath:
          path: /logs/          #node節點目錄
      - name: neusipo-filebeat
        configMap:               #將pod容器內的filebeat配置文件映射到configmap中
          name: neusipo-filebeat    #configmap內filebeat的name
          items:
            - key: filebeat.yml      #映射在configmap中的filebeat的配置文件名
              path: filebeat.yml       #pod中filebeat的配置文件名 

 創建neusipo-filebeat的configmap，映射的文件是從filebeat壓縮包解壓出來的配置文件

[root@iZqj001np4kv147ax0tr54Z ~]# kubectl create configmap neusipo-filebeat --from-file=filebeat.yml -n filebeat
[root@iZqj001np4kv147ax0tr54Z ~]# kubectl get configmap -n filebeat
NAME               DATA   AGE
neusipo-filebeat   1      3h55m   #這里DATA出現1，說明文件已經寫入 

 創建namesapce

[root@iZqj001np4kv147ax0tr54Z ~]# kubectl create configmap ns filebeat
[root@iZqj001np4kv147ax0tr54Z ~]# kubectl get ns |grep filebeat
filebeat              Active   4h25m

 創建filebeat的pod容器，並查看是否創建成功

kubectl apply -f daemonset-demo.yaml -n filebeat

[root@iZqj001np4kv147ax0tr54Z ~]# kubectl get pods -n filebeat
NAME                     READY   STATUS    RESTARTS   AGE
neusipo-filebeat-64mhf   1/1     Running   0          3h55m
neusipo-filebeat-k9ncp   1/1     Running   0          3h55m
neusipo-filebeat-lsvcb   1/1     Running   0          3h55m

 進入容器查看是否掛載上node目錄，並查看是否拉取到微服務的日志

[root@iZqj001np4kv147ax0tr54Z ~]# kubectl exec -it neusipo-filebeat-k9ncp -n filebeat /bin/bash
[root@neusipo-filebeat-k9ncp filebeat-7.13.1-linux-x86_64]# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          40G   17G   21G  45% /
tmpfs            64M     0   64M   0% /dev
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/vda1        40G   17G   21G  45% /sipo-data/neusipo-test
shm              64M     0   64M   0% /dev/shm
tmpfs           7.8G   12K  7.8G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           7.8G     0  7.8G   0% /proc/acpi
tmpfs           7.8G     0  7.8G   0% /proc/scsi
tmpfs           7.8G     0  7.8G   0% /sys/firmware
[root@neusipo-filebeat-k9ncp filebeat-7.13.1-linux-x86_64]# cd /sipo-data/neusipo-test/
[root@neusipo-filebeat-k9ncp neusipo-test]# ls
gateway
[root@neusipo-filebeat-k9ncp neusipo-test]# cd gateway/
[root@neusipo-filebeat-k9ncp gateway]# ls
logs
[root@neusipo-filebeat-k9ncp gateway]# cd logs/
[root@neusipo-filebeat-k9ncp logs]# ls
error  info
[root@neusipo-filebeat-k9ncp logs]# 

 后續只需要更改configmap中的filebeat配置文件，就可以了。

kubectl edit configmap neusipo-filebeat -n filebeat

 完成！！！