logstash簡介
logstash是一個收集日志的組件可以水平伸縮，而且logstash是整個ELK當中擁有最多插件的一個組件，其可以接收來自不同源的數據並統一輸入到指定的且可以是不同目的地。
logstash收集日志基本流程: input-->codec-->filter-->codec-->output
1.input:從哪里收集日志。
2.filter:發出去前進行過濾
3.output:輸出至Elasticsearch或Redis消息隊列
4.codec:輸出至前台，方便邊實踐邊測試
5.數據量不大日志按照月來進行收集
其主要優勢是含有豐富的輸入和輸出格式的支持，其配置格式主要是以下情況：
input { stdin {} } output { elasticsearch { hosts => ["192.168.56.11:9200"] index => "logstash-test-%{+YYYY.MM.dd}" } }
- input 代表輸入源，stdin代表控制台輸入
- output 代表輸出源
詳細配置語法可自行查看博文或者官網
安裝logstash
環境准備：關閉防火牆和Selinux，並且安裝java環境
logstash下載地址：https://artifacts.elastic.co/downloads/logstash/logstash-6.0.0.rpm
[root@linux-node1 ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-6.0.0.rpm
[root@linux-node1 ~]# yum install -y logstash-6.0.0.rpm
[root@linux-node1 ~]# rpm -ql logstash
#node2節點安裝logstash
[root@linux-node2 ~]# yum install -y logstash-6.0.0.rpm
[root@linux-node1 ~]# ll /etc/logstash/conf.d/ #logstash的主配置目錄
總用量 0
logstash的基本語法
input {
指定輸入
}
output {
指定輸出
}
測試標准輸入輸出
[root@linux-node1 ~]# /usr/share/logstash/bin/logstash -e 'input { stdin {} } output { stdout { codec => rubydebug} }' #標准輸入輸出
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
The stdin plugin is now waiting for input:
hello #輸入
{
"@version" => "1", #@version時間版本號，一個事件就是一個ruby對象
"host" => "linux-node1", #host標記事件發生在哪里
"@timestamp" => 2017-12-08T14:56:25.395Z, #@timestamp，用來標記當前事件發生的時間
"message" => "hello" #消息的具體內容
}
測試輸出到文件
[root@linux-node1 ~]# /usr/share/logstash/bin/logstash -e 'input { stdin {} } output { file { path => "/tmp/test-%{+YYYY.MM.dd}.log"} }'
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
hello
[root@linux-node1 ~]# cat /tmp/test-2017.12.09.log
{"@version":"1","host":"linux-node1","@timestamp":"2017-12-09T08:23:14.896Z","message":"hello"}
開啟gzip壓縮輸出
[root@linux-node1 ~]# /usr/share/logstash/bin/logstash -e 'input { stdin {} } outpu{ file { path => "/tmp/test-%{+YYYY.MM.dd}.log.tar.gz" gzip => true } }'
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
hello
[root@linux-node1 ~]# ll /tmp/test-2017.12.09.log.tar.gz
-rw-r--r-- 1 root root 105 12月 9 16:26 /tmp/test-2017.12.09.log.tar.gz
測試輸出到ES
/usr/share/logstash/bin/logstash -e 'input { stdin {} } output { elasticsearch { hosts => ["192.168.56.11:9200"] index => "logstash-test-%{+YYYY.MM.dd}" } }'
指定配置文件啟動
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf -t
配置文件內容
input {
file{
path => "/export/logs/gateway/gateway-provider.%{+YYYY-MM-dd}"
type => "elasticsearch-java-log"
start_position => "beginning"
stat_interval => "2"
codec => multiline {
pattern => "^\[" #以"["開頭進行正則匹配
negate => "true" #正則匹配成功
what => "previous" #和前面的內容進行合並
}
}
}
output {
if [type] == "elasticsearch-java-log" {
elasticsearch {
hosts => ["10.159.42.37:9200"]
index => "gateway-log-%{+YYYY.MM.dd}"
}
}
}
后台運行腳本
nohup /usr/local/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf -w 8 -b 1000 > /dev/null 2>&1 &
參考
https://blog.51cto.com/jinlong/2055424
https://blog.51cto.com/jinlong/2055024
https://blog.51cto.com/jinlong/2056598
https://yq.aliyun.com/articles/604138
https://blog.csdn.net/ljx1528/article/details/100031330
https://my.oschina.net/wangmengjun/blog/861636
https://blog.csdn.net/weixin_34306593/article/details/93020544