實戰一：通過Logstash收集tomcat服務器的日志
1、配置JDK環境
1、解壓JDK包，創建軟鏈接
[root@tomcat-web1 src]# tar xvf jdk-8u212-linux-x64.tar.gz [root@tomcat-web1 src]# ln -sv /usr/local/src/jdk1.8.0_212/ /usr/local/jdk ‘/usr/local/jdk/jdk1.8.0_212’ -> ‘/usr/local/src/jdk1.8.0_212/’ [root@tomcat-web1 src]# ln -sv /usr/local/jdk/bin/java /usr/bin
2、配置java的環境變量
[root@tomcat-web1 ~]# vim /etc/profile.d/jdk.sh #配置環境變量 export HISTTIMEFORMAT="%F %T `whoami`" export export LANG="en_US.utf-8" export JAVA_HOME=/usr/local/jdk export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export PATH=$PATH:$JAVA_HOME/bin [root@tomcat-web1 ~]# . /etc/profile.d/jdk.sh #將環境變量生效
3、查看版本信息及java家目錄信息
[root@tomcat-web1 src]# java -version java version "1.8.0_212" Java(TM) SE Runtime Environment (build 1.8.0_212-b10) Java HotSpot(TM) 64-Bit Server VM (build 25.212-b10, mixed mode) [root@tomcat-web1 src]# echo $JAVA_HOME /usr/local/jdk
2、配置tomcat服務並啟動
1、解壓tomcat服務器的安裝包，並創建tomcat軟鏈接
[root@tomcat-web1 ~]# mkdir /apps [root@tomcat-web1 ~]# cd /apps/ [root@tomcat-web1 apps]# ls apache-tomcat-8.5.42 apache-tomcat-8.5.42.tar.gz tomcat [root@tomcat-web1 apps]# tar xvf apache-tomcat-8.5.42.tar.gz [root@tomcat-web1 apps]# ln -s /apps/apache-tomcat-8.5.42 /apps/tomcat #創建tomcat軟鏈接
2、啟動tomcat服務
[root@tomcat-web1 apps]# /apps/tomcat/bin/startup.sh Using CATALINA_BASE: /apps/tomcat Using CATALINA_HOME: /apps/tomcat Using CATALINA_TMPDIR: /apps/tomcat/temp Using JRE_HOME: /usr/local/jdk Using CLASSPATH: /apps/tomcat/bin/bootstrap.jar:/apps/tomcat/bin/tomcat-juli.jar Tomcat started.
3、設置訪問網站路徑
[root@tomcat-web1 apps]# vim /apps/tomcat/conf/server.xml <Host name="localhost" appBase="/data/tomcat/tomcat_webdir"
4、查看tomcat訪問頁面，此訪問頁面是前面自制的頁面，能訪問，說明tomcat服務正常。
5、修改tomcat服務的配置文件，收集log日志為json格式：/apps/tomcat/bin/server.xml，實際的tomcat訪問log日志文件目錄在：/apps/tomcat/logs/tomcat_access_log.2020-03-13.log
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="tomcat_access_log" suffix=".log" # 將log日志文件名進行修改，以.log為后綴的文件
pattern="{"clientip":"%h","ClientUser":"%l","authenticated":"%u",&quo #修改為json格式日志文件。
t;AccessTime":"%t","method":"%r","status":"%s","SendBytes":"%b","
Query?string":"%q","partner":"%{Referer}i","AgentVersion":"%{User-Agent}i"}"/>
訪問此時的tomcat日志文件：/apps/tomcat/logs/tomcat_access_log.2020-03-13.log ，驗證日志是否是json格式的，可以在網上查看。
[root@logstash ~]# tail -f /apps/tomcat/logs/tomcat_access_log.2020-03-13.log
{"clientip":"192.168.7.1","ClientUser":"-","authenticated":"-","AccessTime":"[13/Mar/2020:15:19:16 +0800]","method":"GET / HTTP/1.1","status":"404","SendBytes":"1078","Query?string":"","partner":"-","AgentVersion":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"}
{"clientip":"192.168.7.1","ClientUser":"-","authenticated":"-","AccessTime":"[13/Mar/2020:15:19:16 +0800]","method":"GET /favicon.ico HTTP/1.1","status":"404","SendBytes":"1078","Query?string":"","partner":"http://192.168.7.102:8080/","AgentVersion":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"}
{"clientip":"192.168.7.1","ClientUser":"-","authenticated":"-","AccessTime":"[13/Mar/2020:15:22:09 +0800]","method":"GET / HTTP/1.1","status":"404","SendBytes":"1078","Query?string":"","partner":"-","AgentVersion":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"}
{"clientip":"192.168.7.1","ClientUser":"-","authenticated":"-","AccessTime":"[13/Mar/2020:15:22:38 +0800]","method":"GET /myapp HTTP/1.1","status":"302","SendBytes":"-","Query?string":"","partner":"-","AgentVersion":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"}
{"clientip":"192.168.7.1","ClientUser":"-","authenticated":"-","AccessTime":"[13/Mar/2020:15:22:38 +0800]","method":"GET /myapp/ HTTP/1.1","status":"200","SendBytes":"14","Query?string":"","partner":"-","AgentVersion":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"}
3、收集tomcat日志文件
需要將logstash主機啟動腳本改為root啟動用戶，否則無法收集到tomcat的日志文件
[root@logstash conf.d]# vim /etc/systemd/system/logstash.service User=root Group=root
重啟logstash服務
[root@logstash conf.d]# systemctl restart logstash
創建/etc/logstash/conf.d目錄下的配置文件：tomcat-java-log.conf
input {
file {
path => "/var/log/logstash/logstash-plain.log" #收集java的日志文件目錄
start_position => "beginning"
stat_interval => 3
type => "java-log"
}
file {
path => "/apps/tomcat/logs/tomcat_access_log.*.log" # 收集tomcat日志的文件目錄
start_position => "beginning"
stat_interval => 3
type => "tomcat-access-log"
codec => "json" # 輸出tomcat 的json日志格式
}
}
output {
if [type] == "java-log" {
elasticsearch {
hosts => ["192.168.7.100:9200"]
index => "javalog-7-102-%{+YYYY.MM.dd}"
}
}
if [type] == "tomcat-access-log" {
elasticsearch {
hosts => ["192.168.7.100:9200"]
index => "tomcat-access-log-7-102-%{+YYYY.MM.dd}"
}
}
}
在kibana網站上創建tomcat日志的索引
在discover選項中，查看添加后的tomcat日志文件
3、收集java日志json格式的文件
1、在/etc/logstash/conf.d/目錄下創建一個java.conf文件，修改此配置文件，將logstash的日志文件收集到elasticsearch主機上。
input {
file {
path => "/var/log/logstash/logstash-plain.log"
start_position => "beginning"
stat_interval => 3
type => "java-log"
}
}
output {
if [type] == "java-log" {
elasticsearch {
hosts => ["192.168.7.100:9200"]
index => "javalog-7-102-%{+YYYY.MM.dd}"
}
}
}
2、重啟logstash服務，並觀察logstash啟動情況
# systemctl restart logstash
在/var/log/logstash/logstash-plain.log文件中可以查看此時的logstash服務器啟動情況。
[root@logstash conf.d]# tail -f /var/log/logstash/logstash-plain.log
[2020-03-13T15:49:04,872][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//192.168.7.100:9200"]}
[2020-03-13T15:49:11,231][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_7d5605c109b000fd1e6e680ae503330d", :path=>["/var/log/logstash/logstash-plain.log"]}
[2020-03-13T15:49:11,291][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_452905a167cf4509fd08acb964fdb20c", :path=>["/var/log/messages"]}
[2020-03-13T15:49:11,297][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_d883144359d3b4f516b37dba51fab2a2", :path=>["/var/log/nginx/access.log"]}
[2020-03-13T15:49:11,387][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x35cee74 run>"}
[2020-03-13T15:49:11,453][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2020-03-13T15:49:11,456][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2020-03-13T15:49:11,478][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2020-03-13T15:49:11,622][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-03-13T15:49:12,874][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} # 查看到此信息，說明logstash已經啟動了。
3、在kibana控制台添加java日志文件
4、此時在discover選項可以看到添加的javalog日志。
5、日志多行合並處理—multiline插件（重點）
目前5.5版本支持此插件，6.x版本已經不需要此版本，可以自動合並日志文件。
官方文檔：https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html
介紹multiline
pattern：正則匹配從哪行合並
negate：true/false，匹配到pattern 部分開始合並，還是不配到的合並
input {
file {
path => "/var/log/logstash/logstash-plain.log" # 要采集的log日志
start_position => "beginning"
codec => multiline {
pattern => "^\[" # 以[開頭開始匹配
negate => true
what => "previous"
}
}
}
output { # 輸出到elasticsearch主機上
elasticsearch {
hosts => ["192.168.7.100:9200"]
index => "logstash-log-7-100-%{+YYYY.MM.dd}"
}}