首先我們用的是elasticsearch+kibana+logstash+filebeat
客戶端filebeat收集日志后經過服務端logstash規則處理后儲存到elasticsearch中,在kibana中展示。
以nginx日志為例
1.我遇到的問題是,logstash中filter的規則似乎未生效,kibana中新建索引總是沒有geoip參數
logstash配置文件如下
input {
beats{
port => 5044
codec => json {
charset => "UTF-8"
}
}
}
filter{
grok {
match => {"message" => '%{DATA:http_x_forwarded_for} - %{DATA:remote_user} \[%{HTTPDATE:time_local}\] "%{DATA:request_uri}"%{NUMBER:status:int} %{NUMBER:body_bytes_sent:int} %{DATA:http_referer} "%{DATA:http_user_agent}"'}
}
if "63nginx_access" in [tags] {
json{
source => "message"
}
if [user_ua] != "-" {
useragent {
target => "agent" #agent將過來出的user agent的信息配置到了單獨的字段中
source => "user_ua" #這個表示對message里面的哪個字段進行分析
}
}
if [http_x_forwarded_for] != "-" {
geoip {
source => "http_x_forwarded_for"
target => "geoip"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}
}
}
}
output {
if[type] == "63nginx_access"{
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "logstash_63nginx_access.%{+YYYY.MM.dd}"
}
}
1.1 創建logstash測試文件用來調試 vim logstash.test.conf
input {
stdin {}
}
filter {
grok {
match => {"message" => '%{DATA:http_x_forwarded_for} - %{DATA:remote_user} \[%{HTTPDATE:time_local}\] "%{DATA:request_uri}"%{NUMBER:status:int} %{NUMBER:body_bytes_sent:int} %{DATA:http_referer} "%{DATA:http_user_agent}"'}
}
if [http_x_forwarded_for] != '-'{
geoip {
source => "http_x_forwarded_for"
target => "geoip"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
}
}
output {
stdout {
codec => rubydebug
}
}
啟動logstash
./bin/logstash -f logstash.test.conf
啟動后粘貼一行nginx的日志
geoip為空,因為我們nginx的http_x_forwarded_for獲取到兩個ip,接着我用單ip測試,一定要是公網ip(內網ip在規則中被過濾了)
啟動logstash
./bin/logstash -f logstash.test.conf
輸入
211.154.222.21 - - [26/Oct/2018:15:07:20 +0800] "GET /pp/index.php?/categories/posted-monthly-list-any-any/start-111210 HTTP/1.0"200 21761 "-""Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
顯然這樣就獲取到geoip的信息了,接着需要調整下nginx日志了
================
nginx日志格式改動牽扯的比較多,還是從logstash中找方法吧
mutate {
split => ["http_x_forwarded_for",","]
add_field => ["real_remote_addr","%{http_x_forwarded_for[0]}"]
}
當http_x_forwarded_for獲取到多個ip時,可以采取以上方式
so我logstash的filter配置文件如下:
filter {
grok {
match => {"message" => '%{DATA:http_x_forwarded_for} - %{DATA:remote_user} \[%{HTTPDATE:time_local}\] "%{DATA:request_uri}"%{NUMBER:status:int} %{NUMBER:body_bytes_sent:int} %{DATA:http_referer} "%{DATA:http_user_agent}"'}
}
mutate {
split => ["http_x_forwarded_for",","]
add_field => ["real_remote_addr","%{http_x_forwarded_for[0]}"]
}
geoip {
source => "real_remote_addr"
target => "geoip"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
}
最后再啰嗦一句
kibana中創建索引一定要以logstash-*開頭,要不kibana中創建地圖時識別不了