Logstash是一個開源的用於收集,分析和存儲日志的工具。
一、環境
# dmidecode|grep "System Information" -A9|egrep "Manufacturer|Product"
Manufacturer: Dell Inc.
Product Name: PowerEdge R630
# uname -a
Linux linux-node2 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
關閉firewalld,selinux
二、安裝Logstash6.2.4
2.1檢查JAVA版本和環境變量
# java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-b10)
OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
# echo $JAVA_HOME
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64
2.2 rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
報錯信息：
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
curl: (6) Could not resolve host: packages.elastic.co; Unknown error
error: https://packages.elastic.co/GPG-KEY-elasticsearch: import read failed(2).
重啟系統成功，有點莫名其妙
添加repo
# vi /etc/yum.repos.d/logstash.repo
[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
# yum install logstash
Loaded plugins: fastestmirror
base | 3.6 kB 00:00:00
centos-ceph-luminous | 2.9 kB 00:00:00
centos-openstack-queens | 2.9 kB 00:00:00
centos-qemu-ev | 2.9 kB 00:00:00
docker-ce-stable | 2.9 kB 00:00:00
elasticsearch-6.x | 1.3 kB 00:00:00
extras | 3.4 kB 00:00:00
logstash-6.x | 1.3 kB 00:00:00
updates | 3.4 kB 00:00:00
(1/4): logstash-6.x/primary | 67 kB 00:00:04
(2/4): centos-ceph-luminous/7/x86_64/primary_db | 115 kB 00:00:04
centos-qemu-ev/7/x86_64/primar FAILED ============================= ] 48 kB/s | 701 kB 00:00:08 ETA
http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/repodata/2dcd3ba7c05dfc6ae2e1da196d1fa38e6f417b3818f3911edf22ddabf779f273-primary.sqlite.bz2: [Errno 14] curl#7 - "Failed to connect to 2605:9000:401:102::2: Network is unreachable"
Trying other mirror.
(3/4): centos-openstack-queens/x86_64/primary_db | 902 kB 00:00:25
centos-qemu-ev/7/x86_64/primar FAILED
http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/repodata/2dcd3ba7c05dfc6ae2e1da196d1fa38e6f417b3818f3911edf22ddabf779f273-primary.sqlite.bz2: [Errno 14] curl#7 - "Failed to connect to 2605:9000:401:102::2: Network is unreachable"
Trying other mirror.
centos-qemu-ev/7/x86_64/primar FAILED
http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/repodata/2dcd3ba7c05dfc6ae2e1da196d1fa38e6f417b3818f3911edf22ddabf779f273-primary.sqlite.bz2: [Errno 14] curl#7 - "Failed to connect to 2605:9000:401:102::2: Network is unreachable"
Trying other mirror.
centos-qemu-ev/7/x86_64/primar FAILED
http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/repodata/2dcd3ba7c05dfc6ae2e1da196d1fa38e6f417b3818f3911edf22ddabf779f273-primary.sqlite.bz2: [Errno 14] curl#7 - "Failed to connect to 2605:9000:401:102::2: Network is unreachable"
Trying other mirror.
centos-qemu-ev/7/x86_64/primar FAILED
http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/repodata/2dcd3ba7c05dfc6ae2e1da196d1fa38e6f417b3818f3911edf22ddabf779f273-primary.sqlite.bz2: [Errno 14] curl#7 - "Failed to connect to 2605:9000:401:102::2: Network is unreachable"
Trying other mirror.
(4/4): centos-qemu-ev/7/x86_64/primary_db | 44 kB 00:00:03
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
logstash-6.x 180/180
Resolving Dependencies
--> Running transaction check
---> Package logstash.noarch 1:6.2.4-1 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
====================================================================================================================================================================
Package Arch Version Repository Size
====================================================================================================================================================================
Installing:
logstash noarch 1:6.2.4-1 elasticsearch-6.x 141 M
Transaction Summary
====================================================================================================================================================================
Install 1 Package
Total download size: 141 M
Installed size: 237 M
Is this ok [y/d/N]: y
Downloading packages:
logstash-6.2.4.rpm | 141 MB 00:10:34
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : 1:logstash-6.2.4-1.noarch 1/1
Using provided startup.options file: /etc/logstash/startup.options
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Successfully created system startup script for Logstash
Verifying : 1:logstash-6.2.4-1.noarch 1/1
Installed:
logstash.noarch 1:6.2.4-1
Complete!]
# systemctl start logstash.service
# systemctl enable logstash.service
#ln -s /usr/share/logstash/bin/logstash /bin/logstash
2.3 測試logstash,
一個日志存儲管道有兩個必需的元素，輸入和輸出，以及一個可選的元素，過濾器。輸入插件使用來自源的數據，過濾器插件根據您的指定修改數據，並且輸出插件將數據寫到目的地。
#/usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2018-05-25 11:30:32.586 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[INFO ] 2018-05-25 11:30:32.617 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[INFO ] 2018-05-25 11:30:32.746 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2018-05-25 11:30:32.782 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2018-05-25 11:30:33.954 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2018-05-25 11:30:34.057 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"935e91a7-5807-460e-9b52-ea8687f18356", :path=>"/usr/share/logstash/data/uuid"}
[INFO ] 2018-05-25 11:30:34.530 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.4"}
[INFO ] 2018-05-25 11:30:34.862 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2018-05-25 11:30:36.620 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
The stdin plugin is now waiting for input:
[INFO ] 2018-05-25 11:30:36.788 [Ruby-0-Thread-1:
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x3cadc2d2@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[INFO ] 2018-05-25 11:30:36.828 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=>1, :pipelines=>["main"]}
helloworld
{
"message" => "helloworld",
"host" => "linux-node1",
"@version" => "1",
"@timestamp" => 2018-05-25T03:31:08.872Z
}
CTRL+D終止PIPE
[INFO ] 2018-05-25 11:32:45.550 [[main]-pipeline-manager] pipeline - Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x3cadc2d2@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
2.4通過Filebeat采集APACHE WEB LOG輸入配置logstash解析，logstash默認已經安裝FILEBEAT。
如果你要發送跨越多行的事件需要在configuration options available in Filebeat中配置而不是使用Multiline codec plugin，使用codec會導致啟動logstash報錯。（比如：多行的Java stack traces消息）
#/usr/share/logstash/bin/logstash-plugin list可查看系統中已默認安裝的插件，如果沒有默認安裝可以進行手動安裝。
手動安裝beat插件：
/usr/share/logstash/bin/logstash-plugin install logstash-input-beats
安裝FILEBEAT
# yum install filebeat -y
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package filebeat.x86_64 0:6.2.4-1 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
====================================================================================================================================================================
Package Arch Version Repository Size
====================================================================================================================================================================
Installing:
filebeat x86_64 6.2.4-1 elasticsearch-6.x 12 M
Transaction Summary
====================================================================================================================================================================
Install 1 Package
Total download size: 12 M
Installed size: 49 M
Downloading packages:
filebeat-6.2.4-x86_64.rpm | 12 MB 00:00:53
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : filebeat-6.2.4-1.x86_64 1/1
Verifying : filebeat-6.2.4-1.x86_64 1/1
Installed:
filebeat.x86_64 0:6.2.4-1
Complete!
2.5配置FILEBEAT yml：
#vim /etc/filebeat/filebeat.yml
filebeat.prospectors:
- type: log
paths:
- /opt/logstash-tutorial.log #這個是下載的APACHE示例日志
output.logstash:
hosts: ["192.168.56.11:5044"]
/usr/share/filebeat/bin/ -e -c /etc/filebeat/filebeat.yml -d "publish" （一直開啟）
Filebeat將嘗試連接端口5044。直到logstack開始使用一個活躍的Beats插件，在這個端口上不會有任何的答案，所以你看到的關於在那個端口上連接失敗的任何消息現在都是正常的。
接下來，創建一個logstash config pipeline，它使用Beats input插件從Beats接收事件。
cat /etc/logstash/first-pipeline.conf
# The # character at the beginning of a line indicates a comment. Use
# comments to describe your configuration.
input {
beats {
port => "5044"
}
}
output {
stdout { codec => rubydebug }
}
#/bin/logstash -f /etc/logstash/first-pipeline.conf --config.test_and_exit
--config.test_and_exit選項將解析配置文件語法並報告錯誤
#/bin/logstash -f /etc/logstash/first-pipeline.conf --config.reload.automatic
--config.reload.automatic 支持自動配置重載，這樣您就不必每次修改配置文件時停止並重新啟動日志存儲
當Logstash啟動后會看驪很多關於
pipelines.yml的警告信息，它是用於一個logstash實例中創建安多個input pipelines
運行LOG：
/bin/logstash -f /etc/logstash/first-pipeline.conf --config.reload.automatic
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2018-05-25 17:08:15.239 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[INFO ] 2018-05-25 17:08:15.256 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[WARN ] 2018-05-25 17:08:16.532 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2018-05-25 17:08:17.162 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.4"}
[INFO ] 2018-05-25 17:08:17.593 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2018-05-25 17:08:19.537 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2018-05-25 17:08:21.015 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2018-05-25 17:08:21.267 [[main]<beats] Server - Starting server on port: 5044
[INFO ] 2018-05-25 17:08:21.308 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x285b71d@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[INFO ] 2018-05-25 17:08:21.599 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=>1, :pipelines=>["main"]}
{
"source" => "/opt/logstash-tutorial.log",
"beat" => {
"version" => "6.2.4",
"name" => "linux-node1",
"hostname" => "linux-node1"
},
"message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"host" => "linux-node1",
"@timestamp" => 2018-05-25T09:12:06.731Z,
"@version" => "1",
"prospector" => {
"type" => "log"
},
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"offset" => 325
}
2.6引入GROK進行WEB日志解析。
# /usr/share/logstash/bin/logstash-plugin install logstash-filter-grok
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Validating logstash-filter-grok
Installing logstash-filter-grok
Installation successful
在first-pipeline.conf中添加grok配置
# vi first-pipeline.conf
input { beats { port => "5044" } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}"} } } output { stdout { codec => rubydebug } }
重新測試先刪除/usr/share/filebeat/bin/data/registry文件，運行結果如下：
{
"agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"verb" => "GET",
"auth" => "-",
"offset" => 325,
"@timestamp" => 2018-05-25T09:53:58.555Z,
"beat" => {
"name" => "linux-node1",
"hostname" => "linux-node1",
"version" => "6.2.4"
},
"clientip" => "83.149.9.216",
"response" => "200",
"host" => "linux-node1",
"message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"bytes" => "203023",
"@version" => "1",
"source" => "/opt/logstash-tutorial.log",
"ident" => "-",
"timestamp" => "04/Jan/2015:05:13:42 +0000",
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"httpversion" => "1.1",
"prospector" => {
"type" => "log"
},
"request" => "/presentations/logstash-monitorama-2013/images/kibana-search.png",
"referrer" => "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\""
}
2.7引入GEOIP插件使解析后的數據更容易搜索，示例中用來搜索IP和地理位置信息
# /usr/share/logstash/bin/logstash-plugin install logstash-filter-geoip #空格只能是一個多了不會運行
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Validating logstash-filter-geoip
Installing logstash-filter-geoip
Installation successful
修改first-pipeline.conf中添加GEOIP配置
# vi first-pipeline.conf
input {
beats {
port => "5044"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
stdout { codec => rubydebug }
}
運行結果
{
"agent" => "\"Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0\"",
"verb" => "GET",
"geoip" => {
"continent_code" => "EU",
"city_name" => "Balham",
"country_code2" => "GB",
"region_code" => "LBH",
"latitude" => 51.4434,
"country_code3" => "GB",
"region_name" => "Lambeth",
"postal_code" => "SW12",
"location" => {
"lat" => 51.4434,
"lon" => -0.1468
},
"ip" => "86.1.76.62",
"timezone" => "Europe/London",
"country_name" => "United Kingdom",
"longitude" => -0.1468
},
"auth" => "-",
"offset" => 24464,
"@timestamp" => 2018-05-25T10:13:03.049Z,
"beat" => {
"name" => "linux-node1",
"hostname" => "linux-node1",
"version" => "6.2.4"
},
"clientip" => "86.1.76.62",
"response" => "200",
"host" => "linux-node1",
"message" => "86.1.76.62 - - [04/Jan/2015:05:30:37 +0000] \"GET /style2.css HTTP/1.1\" 200 4877 \"http://www.semicomplete.com/projects/xdotool/\" \"Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0\"",
"bytes" => "4877",
"source" => "/opt/logstash-tutorial.log",
"@version" => "1",
"ident" => "-",
"timestamp" => "04/Jan/2015:05:30:37 +0000",
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"httpversion" => "1.1",
"prospector" => {
"type" => "log"
},
"request" => "/style2.css",
"referrer" => "\"http://www.semicomplete.com/projects/xdotool/\""
}
2.8添加ELASTICSEARCH中添加INDEX
在上面的測試中已完成了手工測試過程並獲取了我們想得到的數據格式，接下來進行在elasticsearch中添加Index，
現在，web日志被分解成特定的字段， Logstash pipeline可以將數據索引到一個彈性搜索集群中。
修改first-pipeline.conf配置文件，將output設置為
# vi first-pipeline.conf
output {
stdout { codec => rubydebug }
input {
beats {
port => "5044"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => [ "192.168.56.11:9200" ]
}
}
通過這種配置，Logstash使用http協議來連接到Elasticsearch。上面的例子假設在相同的實例上運行了Logstash和Elasticsearch。您可以通過使用主機配置指定類似於hosts => [ "192.168.56.11:9200"之類的東西來指定一個遠程彈搜索實例。
測試運行，先CTRL+C停止運行/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -d "publish"並刪除/usr/share/filebeat/bin/data/registry
# curl -XGET '192.168.56.11:9200/logstash-2018.05.25/_search?pretty&q=response=200'
{
"took" : 2510,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 40,
"max_score" : 0.09844007,
"hits" : [
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "yD_clmMB7ZRkVSxzBppH",
"_score" : 0.09844007,
"_source" : {
"agent" : "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"verb" : "GET",
"geoip" : {
"continent_code" : "EU",
"city_name" : "Moscow",
"country_code2" : "RU",
"region_code" : "MOW",
"latitude" : 55.7485,
"country_code3" : "RU",
"region_name" : "Moscow",
"postal_code" : "101194",
"location" : {
"lat" : 55.7485,
"lon" : 37.6184
},
"ip" : "83.149.9.216",
"timezone" : "Europe/Moscow",
"country_name" : "Russia",
"longitude" : 37.6184
},
"auth" : "-",
"offset" : 3584,
"@timestamp" : "2018-05-25T10:33:05.147Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "83.149.9.216",
"response" : "200",
"host" : "linux-node1",
"message" : "83.149.9.216 - - [04/Jan/2015:05:13:46 +0000] \"GET /presentations/logstash-monitorama-2013/images/Dreamhost_logo.svg HTTP/1.1\" 200 2126 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"bytes" : "2126",
"@version" : "1",
"source" : "/opt/logstash-tutorial.log",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:13:46 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/presentations/logstash-monitorama-2013/images/Dreamhost_logo.svg",
"referrer" : "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\""
}
},
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "yj_clmMB7ZRkVSxzBppH",
"_score" : 0.09844007,
"_source" : {
"agent" : "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"verb" : "GET",
"geoip" : {
"continent_code" : "EU",
"city_name" : "Moscow",
"country_code2" : "RU",
"region_code" : "MOW",
"latitude" : 55.7485,
"country_code3" : "RU",
"region_name" : "Moscow",
"postal_code" : "101194",
"location" : {
"lat" : 55.7485,
"lon" : 37.6184
},
"ip" : "83.149.9.216",
"timezone" : "Europe/Moscow",
"country_name" : "Russia",
"longitude" : 37.6184
},
"auth" : "-",
"offset" : 4234,
"@timestamp" : "2018-05-25T10:33:05.148Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "83.149.9.216",
"response" : "200",
"host" : "linux-node1",
"message" : "83.149.9.216 - - [04/Jan/2015:05:13:46 +0000] \"GET /presentations/logstash-monitorama-2013/images/apache-icon.gif HTTP/1.1\" 200 8095 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"bytes" : "8095",
"@version" : "1",
"source" : "/opt/logstash-tutorial.log",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:13:46 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/presentations/logstash-monitorama-2013/images/apache-icon.gif",
"referrer" : "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\""
}
},
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "0D_clmMB7ZRkVSxzBppH",
"_score" : 0.09844007,
"_source" : {
"agent" : "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"verb" : "GET",
"geoip" : {
"continent_code" : "EU",
"city_name" : "Moscow",
"country_code2" : "RU",
"region_code" : "MOW",
"latitude" : 55.7485,
"country_code3" : "RU",
"region_name" : "Moscow",
"postal_code" : "101194",
"location" : {
"lat" : 55.7485,
"lon" : 37.6184
},
"ip" : "83.149.9.216",
"timezone" : "Europe/Moscow",
"country_name" : "Russia",
"longitude" : 37.6184
},
"auth" : "-",
"offset" : 6167,
"@timestamp" : "2018-05-25T10:33:05.162Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "83.149.9.216",
"response" : "200",
"host" : "linux-node1",
"message" : "83.149.9.216 - - [04/Jan/2015:05:13:47 +0000] \"GET /presentations/logstash-monitorama-2013/css/print/paper.css HTTP/1.1\" 200 4254 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"bytes" : "4254",
"@version" : "1",
"source" : "/opt/logstash-tutorial.log",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:13:47 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/presentations/logstash-monitorama-2013/css/print/paper.css",
"referrer" : "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\""
}
},
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "0T_clmMB7ZRkVSxzBppH",
"_score" : 0.09844007,
"_source" : {
"agent" : "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"verb" : "GET",
"geoip" : {
"continent_code" : "EU",
"city_name" : "Moscow",
"country_code2" : "RU",
"region_code" : "MOW",
"latitude" : 55.7485,
"country_code3" : "RU",
"region_name" : "Moscow",
"postal_code" : "101194",
"location" : {
"lat" : 55.7485,
"lon" : 37.6184
},
"ip" : "83.149.9.216",
"timezone" : "Europe/Moscow",
"country_name" : "Russia",
"longitude" : 37.6184
},
"auth" : "-",
"offset" : 6510,
"@timestamp" : "2018-05-25T10:33:05.162Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "83.149.9.216",
"response" : "200",
"host" : "linux-node1",
"message" : "83.149.9.216 - - [04/Jan/2015:05:13:47 +0000] \"GET /presentations/logstash-monitorama-2013/images/1983_delorean_dmc-12-pic-38289.jpeg HTTP/1.1\" 200 220562 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"bytes" : "220562",
"@version" : "1",
"source" : "/opt/logstash-tutorial.log",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:13:47 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/presentations/logstash-monitorama-2013/images/1983_delorean_dmc-12-pic-38289.jpeg",
"referrer" : "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\""
}
},
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "3D_clmMB7ZRkVSxzBppI",
"_score" : 0.09844007,
"_source" : {
"agent" : "\"Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\"",
"verb" : "GET",
"geoip" : {
"continent_code" : "NA",
"city_name" : "Leander",
"country_code2" : "US",
"region_code" : "TX",
"latitude" : 30.5423,
"country_code3" : "US",
"region_name" : "Texas",
"postal_code" : "78641",
"location" : {
"lat" : 30.5423,
"lon" : -97.9176
},
"ip" : "66.249.73.135",
"timezone" : "America/Chicago",
"country_name" : "United States",
"dma_code" : 635,
"longitude" : -97.9176
},
"auth" : "-",
"offset" : 9295,
"@timestamp" : "2018-05-25T10:33:05.179Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "66.249.73.135",
"response" : "200",
"host" : "linux-node1",
"message" : "66.249.73.135 - - [04/Jan/2015:05:15:03 +0000] \"GET /blog/tags/ipv6 HTTP/1.1\" 200 12251 \"-\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\"",
"bytes" : "12251",
"@version" : "1",
"source" : "/opt/logstash-tutorial.log",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:15:03 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/blog/tags/ipv6",
"referrer" : "\"-\""
}
},
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "4z_clmMB7ZRkVSxzBppI",
"_score" : 0.09844007,
"_source" : {
"agent" : "\"Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0\"",
"verb" : "GET",
"geoip" : {
"continent_code" : "AS",
"city_name" : "Kudus",
"country_code2" : "ID",
"region_code" : "JT",
"latitude" : -6.8048,
"country_code3" : "ID",
"region_name" : "Central Java",
"location" : {
"lat" : -6.8048,
"lon" : 110.8405
},
"ip" : "110.136.166.128",
"timezone" : "Asia/Jakarta",
"country_name" : "Indonesia",
"longitude" : 110.8405
},
"auth" : "-",
"offset" : 10803,
"@timestamp" : "2018-05-25T10:33:05.180Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "110.136.166.128",
"response" : "200",
"host" : "linux-node1",
"message" : "110.136.166.128 - - [04/Jan/2015:05:16:22 +0000] \"GET /favicon.ico HTTP/1.1\" 200 3638 \"-\" \"Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0\"",
"bytes" : "3638",
"@version" : "1",
"source" : "/opt/logstash-tutorial.log",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:16:22 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/favicon.ico",
"referrer" : "\"-\""
}
},
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "5D_clmMB7ZRkVSxzBppI",
"_score" : 0.09844007,
"_source" : {
"agent" : "\"Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0\"",
"verb" : "GET",
"geoip" : {
"continent_code" : "AS",
"city_name" : "Kudus",
"country_code2" : "ID",
"region_code" : "JT",
"latitude" : -6.8048,
"country_code3" : "ID",
"region_name" : "Central Java",
"location" : {
"lat" : -6.8048,
"lon" : 110.8405
},
"ip" : "110.136.166.128",
"timezone" : "Asia/Jakarta",
"country_name" : "Indonesia",
"longitude" : 110.8405
},
"auth" : "-",
"offset" : 11021,
"@timestamp" : "2018-05-25T10:33:05.180Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "110.136.166.128",
"response" : "200",
"host" : "linux-node1",
"message" : "110.136.166.128 - - [04/Jan/2015:05:16:22 +0000] \"GET /images/jordan-80.png HTTP/1.1\" 200 6146 \"http://www.semicomplete.com/projects/xdotool/\" \"Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0\"",
"bytes" : "6146",
"source" : "/opt/logstash-tutorial.log",
"@version" : "1",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:16:22 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/images/jordan-80.png",
"referrer" : "\"http://www.semicomplete.com/projects/xdotool/\""
}
},
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "6T_clmMB7ZRkVSxzBppI",
"_score" : 0.09844007,
"_source" : {
"agent" : "\"-\"",
"verb" : "GET",
"geoip" : {
"continent_code" : "NA",
"country_code2" : "GT",
"region_code" : "HU",
"latitude" : 15.4731,
"country_code3" : "GT",
"region_name" : "Departamento de Huehuetenango",
"location" : {
"lat" : 15.4731,
"lon" : -91.3497
},
"ip" : "200.49.190.101",
"timezone" : "America/Guatemala",
"country_name" : "Guatemala",
"longitude" : -91.3497
},
"auth" : "-",
"offset" : 12114,
"@timestamp" : "2018-05-25T10:33:05.181Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "200.49.190.101",
"response" : "200",
"host" : "linux-node1",
"message" : "200.49.190.101 - - [04/Jan/2015:05:17:39 +0000] \"GET /reset.css HTTP/1.1\" 200 1015 \"-\" \"-\"",
"bytes" : "1015",
"@version" : "1",
"source" : "/opt/logstash-tutorial.log",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:17:39 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/reset.css",
"referrer" : "\"-\""
}
},
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "Az_clmMB7ZRkVSxzBptI",
"_score" : 0.09844007,
"_source" : {
"agent" : "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11\"",
"verb" : "GET",
"geoip" : {
"continent_code" : "EU",
"city_name" : "Paris",
"country_code2" : "FR",
"region_code" : "75",
"latitude" : 48.8574,
"country_code3" : "FR",
"region_name" : "Paris",
"postal_code" : "75011",
"location" : {
"lat" : 48.8574,
"lon" : 2.3795
},
"ip" : "81.220.24.207",
"timezone" : "Europe/Paris",
"country_name" : "France",
"longitude" : 2.3795
},
"auth" : "-",
"offset" : 17730,
"@timestamp" : "2018-05-25T10:33:05.194Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "81.220.24.207",
"response" : "200",
"host" : "linux-node1",
"message" : "81.220.24.207 - - [04/Jan/2015:05:24:57 +0000] \"GET /reset.css HTTP/1.1\" 200 1015 \"http://www.semicomplete.com/blog/geekery/ssl-latency.html\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11\"",
"bytes" : "1015",
"@version" : "1",
"source" : "/opt/logstash-tutorial.log",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:24:57 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/reset.css",
"referrer" : "\"http://www.semicomplete.com/blog/geekery/ssl-latency.html\""
}
},
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "Dz_clmMB7ZRkVSxzBptI",
"_score" : 0.09844007,
"_source" : {
"agent" : "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36\"",
"verb" : "GET",
"geoip" : {
"continent_code" : "NA",
"city_name" : "Denver",
"country_code2" : "US",
"region_code" : "CO",
"latitude" : 39.7313,
"country_code3" : "US",
"region_name" : "Colorado",
"postal_code" : "80218",
"location" : {
"lat" : 39.7313,
"lon" : -104.9692
},
"ip" : "71.212.224.97",
"timezone" : "America/Denver",
"country_name" : "United States",
"dma_code" : 751,
"longitude" : -104.9692
},
"auth" : "-",
"offset" : 20410,
"@timestamp" : "2018-05-25T10:33:05.202Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "71.212.224.97",
"response" : "200",
"host" : "linux-node1",
"message" : "71.212.224.97 - - [04/Jan/2015:05:27:34 +0000] \"GET /reset.css HTTP/1.1\" 200 1015 \"http://www.semicomplete.com/projects/xdotool/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36\"",
"bytes" : "1015",
"@version" : "1",
"source" : "/opt/logstash-tutorial.log",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:27:34 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/reset.css",
"referrer" : "\"http://www.semicomplete.com/projects/xdotool/\""
}
}
]
}
}
查詢可用的INDEX
# curl '192.168.56.11:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open logstash-2018.05.25 Pn0ftjJmTBy4139Os72OuA 5 1 100 0 245.3kb 245.3kb