第十三章·Kibana深入-使用地圖統計客戶端IP

地址庫

在ELK中，我們可以使用地址庫，來對IP進行分析，對日志進行分析，在ELKstack中只有Logstash可以做到，但是出圖，是Kibana來出的，所以我們首先需要下載地址庫數據文件，然后對Logstash進行配置，使用geoip模塊對日志訪問IP進行分析后，再以中國地圖或者是世界地圖的形式，展現在Kibana中。

---

下載地址庫

Logstash2版本下載地址：http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz

logstash5版本下載地址：http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz

#進入Logstash目錄 [root@elkstack03 ~]# cd /etc/logstash/ #下載地址庫 [root@elkstack03 logstash]# wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz #解壓地址庫文件 [root@elkstack03 logstash]# tar xf GeoLite2-City.tar.gz #查看地址庫文件 [root@elkstack03 logstash]# ll 總用量 28784 drwxrwxr-x 2 root root 4096 4月 11 11:36 conf.d drwxr-xr-x 2 2000 2000 4096 4月 8 20:07 GeoLite2-City_20190409 -rw-r--r-- 1 root root 29444833 4月 9 15:32 GeoLite2-City_20190409.tar.gz -rw-rw-r-- 1 root root 1738 3月 23 2017 jvm.options -rw-rw-r-- 1 root root 1334 3月 23 2017 log4j2.properties -rw-rw-r-- 1 root root 4484 3月 5 17:35 logstash.yml -rw-rw-r-- 1 root root 1659 3月 23 2017 startup.options 

配置Logstash使用地址庫

配置Logstash

#進入Logstash配置文件目錄 [root@elkstack03 logstash]# cd /etc/logstash/conf.d/ #編輯Logstash配置文件 [root@elkstack03 conf.d]# vim redis_es_ip.conf input { redis { host => "10.0.0.54" port => "6379" db => "3" key => "all" data_type => "list" password => "zls" } } filter { json { source => "message" remove_field => ["message"] } geoip { source => "clientip" target => "geoip" database => "/etc/logstash/GeoLite2-City_20190409/GeoLite2-City.mmdb" add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ] add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ] } mutate { convert => [ "[geoip][coordinates]", "float"] } } output { elasticsearch { hosts => ["10.0.0.51:9200"] index => "%{type}-%{+YYYY.MM.dd}" } } #啟動Logstash [root@elkstack03 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis_es_ip.conf & #因為是單機環境，日志里面沒有公網IP，所以我們需要自己往里輸入公網IP #北京公網IP [root@elkstack03 conf.d]# echo '{"@timestamp":"2019-04-11T20:27:25+08:00","host":"222.28.0.112","clientip":"222.28.0.112","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"www.elk.com","url":"/index.html","domain":"www.elk.com","xff":"10.0.0.1","referer":"-","status":"304"}' >> /usr/local/nginx/logs/access_json.log #海南公網IP [root@elkstack03 conf.d]# echo '{"@timestamp":"2019-04-11T20:40:24+08:00","host":" 124.225.0.13","clientip":"124.225.0.13","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"www.elk.com","url":"/index.html","domain":"www.elk.com","xff":"10.0.0.1","referer":"-","status":"304"}' >> /usr/local/nginx/logs/access_json.log #吉林公網IP [root@elkstack03 conf.d]# echo '{"@timestamp":"2019-04-11T20:45:24+08:00","host":" 124.234.0.12","clientip":"124.234.0.12","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"www.elk.com","url":"/index.html","domain":"www.elk.com","xff":"10.0.0.1","referer":"-","status":"304"}' >> /usr/local/nginx/logs/access_json.log #黑龍江公網IP [root@elkstack03 conf.d]# echo '{"@timestamp":"2019-04-11T20:46:24+08:00","host":" 123.164.0.18","clientip":"123.164.0.18","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"www.elk.com","url":"/index.html","domain":"www.elk.com","xff":"10.0.0.1","referer":"-","status":"304"}' >> /usr/local/nginx/logs/access_json.log 

---

驗證Kibana中的數據

打開瀏覽器，訪問：http://10.0.0.54:5601

北京公網IP


海南公網IP



吉林公網IP



黑龍江公網IP



配置Kibana使用地圖

Kibana畫中國地圖









如圖：報錯:"No Compatible Fields: The "[www.driverzeng.com -]YYYY.MM.DD" index pattern does not contain any of the following field types: geo_point"

原因：索引格式為[www.driverzeng.com -]YYYY-MM的日志文件由logstash輸出到Elasticsearch；在elasticsearch中，所有的數據都有一個類型，什么樣的類型，就可以在其上做一些對應類型的特殊操作。geo信息中的location字段是經緯度，我們需要使用經緯度來定位地理位置;在elasticsearch中，對於經緯度來說，要想使用elasticsearch提供的地理位置查詢相關的功能，就需要構造一個結構，並且將其類型屬性設置為geo_point，此錯誤明顯是由於我們的geo的location字段類型不是geo_point。

我們可以通過以下方式驗證一下:

[root@elkstack01 ~]# curl -XGET http://10.0.0.51:9200/www.driverzeng.com-2019.04.11/_mapping/ {"www.driverzeng.com-2019.04.11":{"mappings":{"www.driverzeng.com":{"properties":{"@timestamp":{"type":"date"},"@version":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"beat":{"properties":{"hostname":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"version":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"clientip":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"domain":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"geoip":{"properties":{"city_name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"continent_code":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"coordinates":{"type":"float"},"country_code2":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"country_code3":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"country_name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"ip":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"latitude":{"type":"float"},"location":{"type":"float"},"longitude":{"type":"float"},"region_code":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"region_name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"timezone":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"host":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"http_host":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"input_type":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"offset":{"type":"long"},"referer":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"responsetime":{"type":"float"},"size":{"type":"long"},"source":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"status":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"type":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"upstreamhost":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"upstreamtime":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"url":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"xff":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}}} 

其中"location":{"type":"float"},"，字段類型是float，而不是geo_point，因此會報圖中的錯誤。

解決方法：Elasticsearch支持給索引預定義設置和mapping(前提是你用的 elasticsearch 版本支持這個API，不過估計應該都支持)。其實ES中已經有一個默認預定義的模板，我們只要使用預定的模板即可，那為什么還會報錯呢？因為默認預定義的模板必須只有匹配 logstash-* 的索引才會應用這個模板，由於我們在logstash中使用的是[www.driverzeng.com -]YYYY.MM.DD索引方式，因此不會匹配到默認模板，我們只需要改一下索引方式即可:

input {
  redis {
    host => "10.0.0.54" port => "6379" db => "3" key => "all" data_type => "list" password => "zls" } } filter { json { source => "message" remove_field => ["message"] } geoip { source => "clientip" target => "geoip" database => "/etc/logstash/GeoLite2-City_20190409/GeoLite2-City.mmdb" add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ] add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ] } mutate { convert => [ "[geoip][coordinates]", "float"] } } output { elasticsearch { hosts => ["10.0.0.51:9200"] index => "logstash-%{type}-%{+YYYY.MM.dd}" } } 

將輸出到ES的索引: index => "%{type}-%{+YYYY.MM.dd}" 
改為: index => "logstash-%{type}-%{+YYYY.MM.dd}"

重啟Logstash，登錄Kibana刷新即可。

[root@elkstack03 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis_es_ip.conf & 

再次查看Kibana



繼續畫圖



也可以根據自己喜好，畫成熱力圖



保存，可以放入Dashboard