一、設置NGINX日志格式
[root@zabbix_server ~]# vim /etc/nginx/nginx.conf
log_format access_json_log '{"@timestamp":"$time_local",' '"http_host":"$http_host",' '"clinetip":"$remote_addr",' '"request":"$request",' '"status":"$status",' '"size":"$body_bytes_sent",' '"upstream_addr":"$upstream_addr",' '"upstream_status":"$upstream_status",' '"upstream_response_time":"$upstream_response_time",' '"request_time":"$request_time",' '"http_referer":"$http_referer",' '"http_user_agent":"$http_user_agent",' '"http_x_forwarded_for":"$http_x_forwarded_for"}'; access_log /var/log/nginx/access.log access_json_log;
二、在logstash目錄下，下載geolite數據庫。
geoip是logstash的一個過濾插件，用於分析IP獲取地理位置。
root@server-1 logstash]# wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
--2019-11-20 10:23:55-- http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz 正在解析主機 geolite.maxmind.com (geolite.maxmind.com)... 104.17.200.89, 104.17.201.89, 2606:4700::6811:c859, ... 正在連接 geolite.maxmind.com (geolite.maxmind.com)|104.17.200.89|:80... 已連接。 已發出 HTTP 請求，正在等待回應... 200 OK 長度：29963029 (29M) [application/gzip] 正在保存至: “GeoLite2-City.tar.gz” 35% [===========================> ] 10,599,312 24.1KB/s 用時 11m 30s 2019-11-20 10:35:26 (15.0 KB/s) - 在 10599312 字節處連接關閉。重試中。 --2019-11-20 10:35:27-- (嘗試次數： 2) http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz 正在連接 geolite.maxmind.com (geolite.maxmind.com)|104.17.200.89|:80... 已連接。 已發出 HTTP 請求，正在等待回應... 206 Partial Content 長度：29963029 (29M)，剩余 19363717 (18M) [application/gzip] 正在保存至: “GeoLite2-City.tar.gz” 100%[++++++++++++++++++++++++++++====================================================>] 29,963,029 15.2KB/s 用時 9m 9s 2019-11-20 10:44:37 (34.4 KB/s) - 已保存 “GeoLite2-City.tar.gz” [29963029/29963029])
三、解壓
[root@server-1 logstash]# tar -zxvf GeoLite2-City.tar.gz GeoLite2-City_20191119/ GeoLite2-City_20191119/LICENSE.txt GeoLite2-City_20191119/GeoLite2-City.mmdb GeoLite2-City_20191119/COPYRIGHT.txt GeoLite2-City_20191119/README.txt [root@server-1 logstash]#
四、設置logstash配置文件
在/etc/logstash/conf.d目錄下新建一個nginx.conf的配置文件
[root@server-1 conf.d]# vim /etc/logstash/conf.d/nginx.conf
input { beats { port => 10001 } } filter { geoip { source => "clientip" target => "geoip" database => "/etc/logstash/GeoLite2-City_20191119/GeoLite2-City.mmdb" add_field => ["[geoip][coordinates]","%{[geoip][longitude]}"] add_field => ["[geoip][coordinates]","%{[geoip][latitude]}"] } } output { stdout{ codec=>rubydebug } }
source：需要查詢IP位置的源字段
target:目標字段。默認為geoip
database:IP位置信息數據庫目錄
add_field：增加經緯度字段
五、測試配置文件
[root@server-1 conf.d]# logstash -f /etc/logstash/conf.d/nginx.conf WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [INFO ] 2019-11-20 17:17:04.916 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"} [INFO ] 2019-11-20 17:17:04.931 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} [WARN ] 2019-11-20 17:17:05.931 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO ] 2019-11-20 17:17:06.292 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.4"} [INFO ] 2019-11-20 17:17:06.542 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600} [INFO ] 2019-11-20 17:17:08.302 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50} [INFO ] 2019-11-20 17:17:08.329 [[main]-pipeline-manager] geoip - Using geoip database {:path=>"/etc/logstash/GeoLite2-City_20191119/GeoLite2-City.mmdb"} [INFO ] 2019-11-20 17:17:09.704 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:10001"} [INFO ] 2019-11-20 17:17:09.911 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x17715055@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"} [INFO ] 2019-11-20 17:17:09.936 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=>1, :pipelines=>["main"]} [INFO ] 2019-11-20 17:17:09.948 [[main]<beats] Server - Starting server on port: 10001
新開一個SSH連接，查看JAVA進程
[root@server-1 conf.d]# netstat -tunlp|grep java tcp6 0 0 172.28.18.69:9200 :::* LISTEN 18608/java tcp6 0 0 :::10001 :::* LISTEN 16856/java tcp6 0 0 172.28.18.69:9300 :::* LISTEN 18608/java tcp6 0 0 127.0.0.1:9600 :::* LISTEN 16856/java tcp6 0 0 172.28.18.69:9600 :::* LISTEN 15599/java tcp6 0 0 :::514 :::* LISTEN 15599/java
此時，10001端口已經被監聽，啟動成功，過一會屏幕打印收到的NGINX日志數據如如下：
"http_referer" => "http://zabbix.9500.cn/zabbix.php?action=dashboard.view&ddreset=1", "upstream_addr" => "127.0.0.1:9000", "clinetip" => "219.239.8.14", "source" => "/var/log/nginx/access.log", "beat" => { "name" => "zabbix_server.jinglong", "version" => "6.2.4", "hostname" => "zabbix_server.jinglong" }, "fields" => { "log_topics" => "nginx-172.28.18.75" }, "@version" => "1", "upstream_status" => "200", "http_user_agent" => "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.221 Safari/537.36 SE 2.X MetaSr 1.0", "offset" => 20686132, "prospector" => { "type" => "log" }, "request_time" => "0.639", "status" => "200", "host" => "zabbix_server.jinglong" } { "upstream_response_time" => "0.828", "@timestamp" => 2019-11-20T09:31:32.368Z, "http_host" => "zabbix.9500.cn", "tags" => [ [0] "beats_input_raw_event" ], "request" => "GET /map.php?sysmapid=8&severity_min=0&sid=126eba41a3be1fb9&curtime=1574242326679&uniqueid=BCYQV&used_in_widget=1 HTTP/1.1", "http_x_forwarded_for" => "-", "size" => "3502", "geoip" => { "ip" => "219.239.8.14", "longitude" => 116.3883, "country_code2" => "CN", "region_code" => "BJ", "country_code3" => "CN", "continent_code" => "AS", "timezone" => "Asia/Shanghai", "latitude" => 39.9289, "country_name" => "China", "region_name" => "Beijing", "location" => { "lon" => 116.3883, "lat" => 39.9289 } },
此時已經能夠看到geoip的數據了，包括經緯度、國家代碼，國家名稱、城市名稱。
修改配置文件，指定需要的字段
[root@server-1 conf.d]# vim nginx.conf
filter { geoip { source => "clinetip" database => "/etc/logstash/GeoLite2-City_20191119/GeoLite2-City.mmdb" fileds => ["country_name","region_name","longitude","latitude"] } }
fields:指定需要的字段
保存，退出，重新啟動配置文件
"request" => "POST /elasticsearch/_msearch HTTP/1.1", "upstream_status" => "200", "fields" => { "log_topics" => "nginx-172.28.18.75" }, "size" => "24668", "beat" => { "name" => "zabbix_server.jinglong", "hostname" => "zabbix_server.jinglong", "version" => "6.2.4" }, "request_time" => "0.159", "offset" => 20983233, "@version" => "1", "upstream_addr" => "172.28.18.69:5601", "http_host" => "elk.9500.cn" } { "geoip" => { "latitude" => 39.9289, "region_name" => "Beijing", "longitude" => 116.3883, "country_name" => "China" }, "http_user_agent" => "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.221 Safari/537.36 SE 2.X MetaSr 1.0", "prospector" => { "type" => "log" },
此時geoip的數據字段就只顯示我們指定的那幾個了。修改配置文件將數據輸出到elasticsearch
input { beats { port => 10001 } } filter { geoip { source => "clinetip" database => "/etc/logstash/GeoLite2-City_20191119/GeoLite2-City.mmdb" fields => ["country_name","region_name","longitude","latitude"] } } output { elasticsearch { hosts=>["172.28.18.69:9200"] index=>"nginx-172.28.18.75-%{+YYYY.MM.dd}" } }
啟動logstash配置文件nginx.conf
~ [root@server-1 conf.d]# logstash -f /etc/logstash/conf.bak/nginx.conf WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [INFO ] 2019-11-21 09:00:40.934 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"} [INFO ] 2019-11-21 09:00:40.965 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} [WARN ] 2019-11-21 09:00:41.962 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO ] 2019-11-21 09:00:42.365 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.4"} [INFO ] 2019-11-21 09:00:42.637 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600} [INFO ] 2019-11-21 09:00:44.436 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50} [INFO ] 2019-11-21 09:00:45.078 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://172.28.18.69:9200/]}} [INFO ] 2019-11-21 09:00:45.089 [[main]-pipeline-manager] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://172.28.18.69:9200/, :path=>"/"} [WARN ] 2019-11-21 09:00:45.337 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://172.28.18.69:9200/"} [INFO ] 2019-11-21 09:00:45.856 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>6} [WARN ] 2019-11-21 09:00:45.857 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6} [INFO ] 2019-11-21 09:00:45.874 [[main]-pipeline-manager] elasticsearch - Using mapping template from {:path=>nil} [INFO ] 2019-11-21 09:00:45.878 [[main]-pipeline-manager] elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}} [INFO ] 2019-11-21 09:00:45.897 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//172.28.18.69:9200"]} [INFO ] 2019-11-21 09:00:45.902 [[main]-pipeline-manager] geoip - Using geoip database {:path=>"/etc/logstash/GeoLite2-City_20191119/GeoLite2-City.mmdb"} [INFO ] 2019-11-21 09:00:46.712 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:10001"} [INFO ] 2019-11-21 09:00:46.846 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x1b610349@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"} [INFO ] 2019-11-21 09:00:46.909 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=>1, :pipelines=>["main"]} [INFO ] 2019-11-21 09:00:46.911 [[main]<beats] Server - Starting server on port: 10001
六、配置kibana展示
打開kibana,建立索引
下一步，點擊創建索引模式，創建成功后，就能看到索引對應的字段列表，其中包含geoip字段
在“發現“里，新建，選擇剛才建立的索引模式，，此時能看到關於geoip的相關字段
接下來，用地圖展示數據
“可視化”里面點擊創建一個可視化視圖“，選擇“坐標地圖”
選擇創建的索引，選擇“選擇buckets類型”為"GEOHASH"
此時報錯：
說沒有發現字段類型為geo_point的數據字段,此時需要修改logstash配置文件，增加location字段
input { beats { port => 10001 } } filter { geoip { source => "clinetip" database => "/etc/logstash/GeoLite2-City_20191119/GeoLite2-City.mmdb" fields => ["country_name","region_name","location"] } } output { elasticsearch { hosts=>["172.28.18.69:9200"] index=>"nginx-172.28.18.75-%{+YYYY.MM.dd}" } }
重啟logstash配置文件,並刪除elasticsearch 的索引
[root@server-1 conf.d]# curl -XDELETE http://172.28.18.69:9200/nginx-172.28.18.75-*
重啟kibana
root@server-1 conf.d]# systemctl restart kibana
打開kibana,重新建立索引，發現已經有了geoip.location字段
再建立坐標地圖,還是報錯
后來，百度發現是因為輸出index的文件名不對，必須以logstash開頭才可以使location字段輸出為geo_point類型，於是修改logstash配置文件
input { beats { port => 10001 } } filter { geoip { source => "clinetip" database => "/etc/logstash/GeoLite2-City_20191119/GeoLite2-City.mmdb" fields => ["country_name","region_name","location"] #add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ] #add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ] } } output { elasticsearch { hosts=>["172.28.18.69:9200"] index=>"logstash-nginx-172.28.18.75-%{+YYYY.MM.dd}" } }
將index文件名改為logstash-nginx-172.28.18.75-%{+YYYY.MM.dd}，重新啟動配置文件，並刪除以前的index
[root@server-1 conf.d]# logstash -f /etc/logstash/conf.bak/nginx.conf
curl -XDELETE http://172.28.18.69:9200/nginx-172.28.18.75-2019.11.21
打開kibana，刪除之前的索引，重新建立索引
此時，發現geoip.location字段的類型變成了geo_point，問題解決，重新建立坐標地圖
展示數據成功。
七、使用高德地圖展示數據為中文
編輯kibana配置文件，在最后加一行
tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'
[root@server-1 conf.d]# vim /etc/kibana/kibana.yml
# The default locale. This locale can be used in certain circumstances to substitute any missing # translations. #i18n.defaultLocale: "en" tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'
重啟kibana
[root@server-1 conf.d]# systemctl restart kibana
刷新kibana頁面，即可顯示中文地圖
