[elk]logstash的grok匹配邏輯grok+date+mutate

重點參考:
http://blog.csdn.net/qq1032355091/article/details/52953837

logstash的精髓:

grok插件原理
date插件原理
kv插件原理

日志默認情況

默認將日志內容賦給了message字段, logstash附加了@timestamp @version host 3個字段

{
    "@timestamp" => 2017-11-30T06:09:09.625Z,
      "@version" => "1",
          "host" => "lb-212-222.above.com",
       "message" => "sad"
}

match匹配原則

參考: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

date插件匹配過程解析

input { stdin { } }
 
filter {
  grok { match => [ "message", "%{HTTPDATE:[@metadata][timestamp]}" ] }
  date { match => [ "[@metadata][timestamp]", "dd/MMM/yyyy:HH:mm:ss Z" ] }
}
 
output {
  stdout { codec => rubydebug }
}

##用正則HTTPDATE匹配message,將結果賦給[@metadata][timestamp]字段
grok { match => [ "message", "%{HTTPDATE:[@metadata][timestamp]}" ] }

##date插件將[@metadata][timestamp]的值賦給 @timestamp字段
date { match => [ "[@metadata][timestamp]", "dd/MMM/yyyy:HH:mm:ss Z" ] }

下面是一個完整例子:
參考: http://blog.csdn.net/xiaoyu_bd/article/details/52531051

input  {
     stdin{}
}
filter {
    grok {
        match => ["message", "%{TIMESTAMP_ISO8601:logdate}"]
    }
    date {
        match => ["logdate", "yyyy-MM-dd HH:mm:ss,SSS"]
        target => "@timestamp"  ## 默認target就是"@timestamp
    }
}
output{
    stdout{
        codec=>rubydebug{}
    }
}

date {
    match => [“timestamp”, “dd/MMM/yyyy:HH:mm:ss Z”]
    #默認目標就是@timestamp
    target => "@timestamp"
    "locale" => "en"
}

mutate插件

• 修改字段類型
  參考(修改時間格式): http://blog.csdn.net/wang_zhenwei/article/details/49760975

  mutate {  
  
      convert => { "dest_Port" => "integer" }  
      convert => { "source_Port" => "integer" }  
   }  

• 添加字段

input { stdin { } }
 
filter {
  mutate { add_field => { "show" => "This data will be in the output" } }
}
 
output {
    stdout { codec => rubydebug }
}

• 還可以轉換字段大小寫

kibana 查詢結果csv導出

table類型的導出:

餅圖統計結果導出