4.安裝fluentd用於收集集群內部應用日志

作者

微信:tangy8080
電子郵箱:[e-mail]
更新時間：2019-06-13 11:02:14 星期四

歡迎您訂閱和分享我的訂閱號,訂閱號內會不定期分享一些我自己學習過程中的編寫的文章
如您在閱讀過程中發現文章錯誤,可添加我的微信 tangy8080 進行反饋.感謝您的支持。

文章主題

• 在大多數情況下,我們需要集中管理應用的日志.但是我們又不能強制要求開發者直接對日志進行統一輸出
  對開發者來說這可能是侵入式的,為了統一輸出日志,可能導致業務收到影響.
  在這種情況下我們可以自己采集日志,采集工具比較多,這里我們選擇fluentd,用於收集集群內的日志
• 集群外部日志的收集將在下一節涉及
• 日志收集原則:在任意時刻(由Es高性能索引速度支持)可以實時的查看到某一節點(由physics.hostname支持)的某一服務(由tag支持)的日志

前置條件

• 已經完成了本章節的第一，第二節

安裝

開始安裝fluentd-elasticsearch

由於我們對fluentd-elasticsearch的定制比較多,所以我們選擇使用源碼來安裝fluentd-elasticsearch

#克隆源碼庫
cd /usr/local/src/
git clone https://github.com/kiwigrid/helm-charts
cd helm-charts/charts/fluentd-elasticsearch/

#為所有輸出添加hostname,physics.hostname,tag字段,方便查找日志
vim /usr/local/src/helm-charts/charts/fluentd-elasticsearch/templates/configmaps.yaml
#在containers.input.conf節點下添加如下配置
    <filter **>
      @id filter_hostname
      @type record_transformer
      <record>
        hostname "#{Socket.gethostname}"
        physics.hostname "#{ENV['K8S_NODE_NAME']}"
        tag ${tag}
      </record>
    </filter>

• 在filter中添加了hostname,該值為k8s中的podName
• physics.hostname標識節點的名稱,該值為k8s中節點的名稱,在daemonset.yaml中定義默認定義了K8S_NODE_NAME名稱

#執行安裝
cd /usr/local/src/helm-charts/charts/fluentd-elasticsearch

helm install --name fluentd-elasticsearch --set elasticsearch.host=elasticsearch-client,hostLogDir.dockerContainers=/data/k8s/docker/data/containers .

• hostLogDir.dockerContainers 用於收集docker產生的日志,請按實際情況更改
• stable/fluentd-elasticsearch is deprecated as we move to our own repo (https://kiwigrid.github.io) which will be puplished on hub.helm.sh soon. The chart source can be found here: https://github.com/kiwigrid/helm-charts/tree/master/charts/fluentd-elasticsearch

[按需]卸載fluentd-elasticsearch

helm del --purge fluentd-elasticsearch

采集思路解析

• 我們在每個節點上運行一個fluentd代理（采用DaemonSet）,用於采集部署在k8s容器環境中程序和部署在物理主機中程序的日志。
• 對於容器內部應用.它的生命周期是不固定的,所以我們可以把日志放在宿主機上,（也可以使用sidecar容器，https://kubernetes.io/docs/concepts/cluster-administration/logging/）,出於性能原因,這里會選擇只運行一個fluentd代理)
  
• pos_file需要放在何處? 需要放在宿主機上的一個穩定位置（通常和設定和path一個目錄）.它記錄了采集進度,假如容器銷毀重啟后,它依然可以從上次進度的位置開始采集
  
• 日志輪轉問題, 如果應用本身不支持日志輪轉.我們需要考慮日志輪轉問題.避免日志越來越大

采集實例配置

實例一:運行在物理機器中的Nginx日志

配置掛載目錄

vim /usr/local/src/helm-charts/charts/fluentd-elasticsearch/values.yaml

#nginx日志不在/var/log目錄下,在文件中最后添加Nginx日志文件掛載
extraVolumes:
   - name: nginxlog
     hostPath:
      path: /usr/local/nginx/logs
          
extraVolumeMounts:
   - name: nginxlog
     mountPath: /var/log/nginx
     readOnly: true

配置采集數據源

vim /usr/local/src/helm-charts/charts/fluentd-elasticsearch/templates/configmaps.yaml

    # service.honeysuckle-log-consumer
    <source>
      @id honeysuckle-log-consumer.log
      @type tail
      <parse>
        @type regexp
        expression /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?<thread>.*) (?<level>[a-zA-Z]+) (?<logger>.*) (?<property>\[.*\]) - (?<msg>[\s\S]*)$/ 
        time_key time
      </parse>
      path /var/log/businesslogs/honeysuckle-log-consumer/Log.txt
      pos_file /var/log/businesslogs/honeysuckle-log-consumer/Log.txt.pos
      tag service.honeysuckle-log-consumer
    </source>
    
    
    # service.honeysuckle-configmanager-service
    <source>
      @id honeysuckle-configmanager-service.log
      @type tail
      <parse>
        @type regexp
        expression /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?<thread>.*) (?<level>[a-zA-Z]+) (?<logger>.*) (?<property>\[.*\]) - (?<msg>[\s\S]*)$/ 
        time_key time
      </parse>
      path /var/log/businesslogs/honeysuckle-configmanager-service/Log.txt
      pos_file /var/log/businesslogs/honeysuckle-configmanager-service/Log.txt.pos
      tag service.honeysuckle-configmanager-service
    </source>
    
    # Nginx Access Log Source
    <source>
      @id nginx.accesslog
      @type tail
      path /var/log/nginx/access.log
      pos_file /var/log/access.log.pos
      format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)"(?:\s+(?<http_x_forwarded_for>[^ ]+))?)?$/
      time_format %d/%b/%Y:%H:%M:%S %z
      tag nginx
     </source>

• nginx默認的日志格式format可以在這里找到
  https://docs.fluentd.org/parser/nginx

實例二:運行在容器中的Helloword程序

該項目用於模擬一個服務,它每隔3秒向 /app/App_Data/Logs/Log.txt 寫入一條日志
該項目采用Log4net寫入日志,應用本身支持日志輪轉,最新的日志都在Log.txt中
log4net.config的配置如下:

<?xml version="1.0" encoding="utf-8" ?>
<log4net>
  <!--文本文件appender-->
  <appender name="RollingFileAppender" type="log4net.Appender.RollingFileAppender" >
    <file value="App_Data/Logs/Log.txt" />
    <appendToFile value="true" />
    <rollingStyle value="Size" />
    <maxSizeRollBackups value="10" />
    <maximumFileSize value="1024KB" />
    <staticLogFileName value="true" />
    <layout type="log4net.Layout.PatternLayout">
      <conversionPattern value="%date [%thread] %-5level %logger [%property{NDC}] - %message%newline" />
    </layout>
  </appender>

  <root>
    <appender-ref ref="RollingFileAppender" />
    <level value="ALL" />
  </root>
</log4net>

日志輸出的樣例類似於:

2019-06-17 07:29:52,038 [5] INFO  honeysuckle.log.consumer.HoneysuckleWebModule [(null)] - electronicinvoice_log_queue.BasicConsume 已創建.
2019-06-17 07:31:51,510 [WorkPool-Session#1:Connection(21956434-0138-45f3-be65-848ca544cad3,amqp://honeysuckle.site:5672)] ERROR honeysuckle.log.consumer.HoneysuckleWebModule [(null)] - Error in EventingBasicConsumer.Received
Elasticsearch.Net.UnexpectedElasticsearchClientException: The operation was canceled. ---> System.OperationCanceledException: The operation was canceled.
   at System.Net.Http.HttpClient.HandleFinishSendAsyncError(Exception e, CancellationTokenSource cts)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
   at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
   at Nest.LowLevelDispatch.BulkDispatch[TResponse](IRequest`1 p, SerializableData`1 body)
   at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.Dispatch[TRequest,TQueryString,TResponse](TRequest request, Func`3 responseGenerator, Func`3 dispatch)
   at honeysuckle.log.consumer.HoneysuckleWebModule.<>c__DisplayClass10_0.<CreateNewChannel>b__0(Object sender, BasicDeliverEventArgs e) in /src/src/honeysuckle.log.consumer/HoneysuckleWebModule.cs:line 141

基於日志輸出格式,我們采用正則表達式插件進行日志解析（表達式在下面的source中可以看到）
可以使用fluentular工具進行表達式的正確性測試

該項目托管在:

http://[e-mail]/r/public/helloworld.git

如有測試需要.歡迎clone

配置采集數據源

vim /usr/local/src/helm-charts/charts/fluentd-elasticsearch/templates/configmaps.yaml

    # service.helloworld.log Log Source
    <source>
      @id helloworld.log
      @type tail
      <parse>
        @type regexp
        expression /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?<thread>\[\d+\]) (?<level>[a-zA-Z]+) (?<logger>.*) (?<property>\[.*\]) - (?<msg>.*)$/
        time_key time
      </parse>
      path /var/log/businesslogs/helloworld/Log.txt
      pos_file /var/log/businesslogs/helloworld/Log.txt.pos
      tag service.helloworld
    </source>

查看fluentd是否采集到了數據

待所有組件全部runing之后,可以通過curl查看fluentd是否采集到了數據

curl 'http://10.254.193.78:9200/_cat/indices?v'
health status index               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   logstash-2019.06.11 _v7q6b4DSt6mLJ_GLDyFUQ   5   1       4112            0      4.6mb          2.2mb
yellow open   logstash-2019.06.12 mz3sF15LTbqQwZiXMubdgg   5   1      29394            0     27.9mb         15.2mb
green  open   logstash-2019.06.13 rtjvfGVWSM-5rLvq72jxlA   5   1        971            0      2.4mb          1.1mb

在Kibana中檢查日志收集情況

• 檢查點一:日志是否根據tag正確分類
  

• 檢查點二：日志否則附加了hostname和physics.hostname屬性
  

處理升級問題

隨着業務的需要,我們后期可能會加入其他的服務.這個時候我們可以在配置文件（configmaps.yaml）中添加好采集規則之后,進行升級

cd /usr/local/src/helm-charts/charts/fluentd-elasticsearch
helm upgrade fluentd-elasticsearch .

一些問題的處理方式記錄

• failed to read data from plugin storage file path="/var/log/kernel.pos/worker0/storage.json" error_class=Fluent::ConfigError error="Invalid contents (not object) in plugin storage file: '/var/log/kernel.pos/worker0/storage.json

rm /var/log/kernel.pos/worker0/storage.json

引用鏈接

https://github.com/helm/charts/tree/master/stable/kibana#configuration