【Logstash系列】使用Logstash作為收集端采集IIS日志

現階段Logstash在Windows端的日志采集一直存在若干問題，包括：

 

1. LS有讀鎖：進程開啟后Input指定路徑下的所有文件都會被鎖死無法重命名或刪除。

2. LS不識別*：如果在path上使用“*”的話，sincedb會失效，所有日志都會從文件開始處重新讀取（類似start_position => beginning）。

 

社區​里關於這些歷史問題也是一抓一大把，貼下主要的幾個貼：

[ LOGSTASH-429] File Input - .sincedb file is broken on Windows

[ LOGSTASH-578] Logstash Agent lock file inputs on Windows - Python LOG rotate failed

[ LOGSTASH-986] File Locking in Windows(Rolling logs, etc.)

[ LOGSTASH-1587] Windows - file input - sincedb / start_position is ignored

[ LOGSTASH-1801] Input File Rotation in Logstash

 

當然，開發者也一直在努力修復這些問題，這兩個月貌似有了不少進展，Jordan並號稱LS1.5將會修復這個BUG，那就讓我們歡呼等待吧:)

Release a gem with windows support  #39

File Locking in Windows(Rolling logs, etc.)  #1557

 

那么，問題來了，在現有的LS1.4.2版本上，如何在生產環境上對日志進行采集呢？

 

阿婆主采用的辦法是每周日凌晨執行計划任務，關閉LS服務，刪除除了當天以前其他所有的IIS日志，然后再啟動。這樣就保證了IIS日志不會無法被刪除，且重啟后即使重復讀也不會重復讀太多文件而影響性能。

不過這樣部署起來的成本比較大，需要一台台上去裝啊改的，覺得麻煩的話還是考慮下NXLOG方式進行采集，之后阿婆主也會提到如何使用nxlog來采集IIS日志。

 

***小小分割線***

 

以下附上配置文檔（安裝部署文檔本篇暫不涉及，需要的話可私信）：

• LS客戶端配置文件

input中使用file及eventlog模塊分別對IIS及Windows Event Log進行采集；filter中用grok對日志進行分隔處理；output將過濾后的日志傳至redis。

input {

  file {

    type => "IIS"

    path => "D:\iislog\xxx.xxx.com\W3SVC663037409/*.log"

    codec => plain {

charset => "ISO-8859-1"

    }

  }

}

 

input {

  eventlog {

    type => 'Win32-EventLog'

    logfile => ["Application", "System", "Security"]

  }

}

 

filter {

  #ignore log comments

  if [message] =~ "^#" {

    drop {}

  }

 

  grok {

    # check that fields match your IIS log settings

    match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} (%{WORD:s-sitename}|-) (%{IPORHOST:s-ip}|-) (%{WORD:cs-method}|-) %{NOTSPACE:cs-uri-stem} %{NOTSPACE:cs-uri-query} (%{NUMBER:s-port}|-) (%{IPORHOST:c-ip}|-) %{NOTSPACE:cs-useragent} %{NOTSPACE:cs-referer} %{NOTSPACE:cs-host} (%{NUMBER:sc-status}|-) (%{NUMBER:sc-bytes}|-) (%{NUMBER:cs-bytes}|-) (%{NUMBER:time-taken}|-)"]

  }

  #Set the Event Timesteamp from the log

date {

    match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]

 timezone => "Etc/UCT"

  }

  mutate {

remove_field => [ "log_timestamp"]

convert => [ "sc-bytes", "float" ]

convert => [ "cs-bytes", "float" ]

convert => [ "time-taken", "float" ]

  }

}

 

output {

    #stdout { codec => rubydebug }

    redis {

    host => "192.168.xx.xxx"

    data_type =>"list"

    key => "test:redis"

   }

}

注意：

1. charset是為了解決logstash字符集的問題，如果按默認值“UTF-8”可能還是無法識別一些特殊字體，而修改為“ISO-8859-1”后便可以正確的讀入了：

←[33mReceived an event that has a different character encoding than you configured. {:text=>"2014-12-22 14:22:52 /newDict/jp/improve_new.aspx sourcedict=1&jid=322316&wtype=jc&w=\\xCD?\\xFC 192.168.31.190 HTTP/1.1 Mozilla/4.0 - dict.hjenglish.com 200 5043 342\\r", :expected_charset=>"UTF-8", :level=>:warn}←[0m[33mInterrupt received. Shutting down the pipeline. {:level=>:warn}←[0m

2. 如果要調用eventlog模塊，必須安裝“ contrib plugins”，否則服務無法啟動:

報錯：“LoadError: no such file to load -- jruby-win32ole”

3. 針對數值型字段，必須使用mutate中的convert強制轉換字段類型，否則還是字符型，ES后期無法對其使用聚集方法（mean/total/max/etc.）

報錯：

ELK日志錯誤：

[2015-01-07 11:43:02,464][DEBUG][action.search.type       ] [Prester John] [logstash-2015.01.07][0], node[wL6TfFyxQI2fmsDWHI-bdA], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@61fc4bf9] lastShard [true]

org.elasticsearch.search.SearchParseException: [logstash-2015.01.07][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"facets":{"0":{"date_histogram":{"key_field":"@timestamp","value_field":"time-taken","interval":"1s"},"global":true,"facet_filter":{"fquery":{"query":{"filtered":{"query":{"query_string":{"query":"cs-host:(dict.*)"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1420601882243,"to":1420602182243}}},{"terms":{"s-ip.raw":["192.168.33.31"]}}]}}}}}}}},"size":0}]]

        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)

        at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)

        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)

        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)

        at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)

        at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)

        at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.ClassCastException: org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData

        at org.elasticsearch.search.facet.datehistogram.DateHistogramFacetParser.parse(DateHistogramFacetParser.java:199)

        at org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93)

        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)

        ... 9 more

未使用convert前：

字段類型轉換后：

4. 對於無法正確獲得客戶端IP而取到了前端HAProxy或F5或CDN的IP地址的話，需要使用“X-Forwarded-For”，可以看下F5官方提供的文檔：

https://devcentral.f5.com/articles/x-forwarded-for-log-filter-for-windows-servers

 

5. 如何對grok表達式進行調試，三斗室大大也早有過介紹了，另外評論區1樓有彩蛋：

http://chenlinux.com/2014/10/19/grokdebug-commandline/

 

6. IIS日志域必須與grok中的配置項精確匹配，否則會出現grok匹配失敗，返回“_grokparsefailure”的情況：

補充一下：在這里如何使用powershell對IIS日志域進行批量更改，阿婆主研究了一下午還是沒弄出來，希望有相關經驗的朋友給予下幫助哈~

 

• LS服務端配置

input中用到redis模塊導入日志，output中用elasticsearch模塊將消息導至ES。

input {

  redis {

    host => '192.168.xx.xxx'

    data_type => 'list'

    port => "6379"

    key => 'test:redis'

    type => 'redis-input'

        }

}

output {

  elasticsearch {

    host => "192.168.xx.xxx"

    port => "9300"

                }

}

 

• 批處理腳本導入計划任務

addTask.bat

schtasks /create /tn "LSRestartIISDel" /tr D:\scripts\logstash\LSRestartIISDel.bat /sc weekly /mo 1 /d SUN /st 03:00 /ru system

pause

注意：Windows Server 2003需要安裝 補丁（需重啟），否則只能手動添加。

 

• 批處理腳本每周刪除IIS日志

LSRestartIISDel.bat

:: Due to LogStash Bug, LS process will lock IIS file until process stopped, however IIS log need to be deleted everyday, so we need stop LS, delete the log and start LS manually.

:: Create Date#2015-01-07#zhoufeng

:: Ver1.0

 

@echo off

 

:: stop LS Service

NET STOP logstash

 

:: delete iis log before 1 days

:: CHANGE IIS LOG PATH BEFORE USING IT!!

forfiles -p "D:\iislog\xxx.xxx.com\W3SVC663037409" /s /m *.log /c "cmd /c del @path" /d -1

 

:: start service

NET START logstash

注意：刪除的路徑必須手動修改！

 

***神秘分割線***

 

參考文檔：

[ Link] Using Logstash to Analyse IIS Log Files with Kibana

[ Link] Logstash: Received an event that has a different character encoding

[ Link] Win32-EventLog input throwing "LoadError: no such file to load -- jruby-win32ole"

[ Link] Kibana 中文指南 - histogram by 三斗室

[ Link] chenryn/logstash-best-practice-cn/input/file.md by 三斗室