Logstash收集nginx日志之使用grok過濾插件解析日志

grok作為一個logstash的過濾插件，支持根據模式解析文本日志行，拆成字段。

• nginx日志的配置：

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for"';

 

• logstash中grok的正則（添加在logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-xxx/patterns/grok-patterns文件中）為：

WZ ([^ ]*)
NGINXACCESS %{IP:remote_ip} \- \- \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{WZ:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:status} %{NUMBER:bytes} %{QS:referer} %{QS:agent} %{QS:xforward}

 

logstash的配置為：

input {
    file {
        path     => ["/var/log/nginx/access.log"]
        type    => "nginxlog"
        start_position => "beginning"
    }
}

filter {  
    grok {  
      match => { "message" => "%{NGINXACCESS}" }
    }  
} 
output {
    stdout {
        codec    => rubydebug
    }
}

logstash的輸出：

{
        "message" => "192.168.154.2 - - [30/Mar/2017:01:27:09 -0700] \"GET /index.html HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36\" \"-\"",
       "@version" => "1",
     "@timestamp" => "2017-03-30T08:27:09.539Z",
           "path" => "/var/log/nginx/access.log",
           "host" => "spark4",
           "type" => "nginxlog",
      "remote_ip" => "192.168.154.2",
      "timestamp" => "30/Mar/2017:01:27:09 -0700",
         "method" => "GET",
        "request" => "/index.html",
    "httpversion" => "1.1",
         "status" => "304",
          "bytes" => "0",
        "referer" => "\"-\"",
          "agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36\"",
       "xforward" => "\"-\""
}