nginx-access.log的logstash解析

1、nginx的access的格式：

log_format hehe '$http_host $server_addr $remote_addr [$time_local] "$request" $request_body $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_time $upstream_response_time $upstream_addr';

 

2、nginx日志

evo.xinpinget.com 10.1.217.143 6.52.65.86 [22/May/2018:15:10:46 +0800] "GET /api/merchant/listForSelection?filterByUserSelector=true&token=1cmqfl2RiO9BV9xIU8F3LG1I HTTP/1.1" - 200 268 "http://evo.xinpinget.com/order/newAfterSales" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36 QQBrowser/4.1.4132.400" 0.021 0.021 10.1.132.201:4010

 

 

3、logstash

patterns：

# cat nginx

HOSTPORT1 (%{IPV4}:%{POSINT}[, ]{0,2})+
NGINXACCESS1 %{IPORHOST:http_host} %{IPORHOST:server_addr} %{IPORHOST:remote_addr} \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATH:uri}\?%{NOTSPACE:params} HTTP/%{NUMBER:http_version}" %{DATA:request_body|-} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent} (%{NUMBER:request_time:float}|-) (%{NUMBER:upstream_response_time:float}|-) %{HOSTPORT1:upstream_addr}

 

logstash.conf

filter { 
 if 'api' in [tags] {
                  grok {
                    patterns_dir => "/etc/logstash/patters.d"
                    match => { "message" => "%{NGINXACCESS1}" }
                  }
                  kv {
                source => "params"
                field_split => "&?"
                value_split => "="
                include_keys => [ "token","user" ]
      }
        }
}