ELK(8)：ELK-logstash收集日志寫入數據庫
在使用ELK對日志進行收集的時候，如果需要對數據進行存檔，可以考慮使用數據庫的方式
其實我不建議，日志真的太多了，數據庫扛不住的
安裝logstash的數據庫插件
安裝logstash的數據庫插件需要先安裝gem源:
sudo yum install gem -y sudo gem –v #替換gem源 gem source list sudo gem sources --add https://gems.ruby-china.com/ --remove https://rubygems.org/ #查看當前已經安裝的插件： /usr/share/logstash/bin/logstash-plugin list
#安裝
sudo /usr/share/logstash/bin/logstash-plugin install logstash-output-jdbc
[admin@pe-jira gems]$ sudo /usr/share/logstash/bin/logstash-plugin install logstash-output-jdbc Validating logstash-output-jdbc Installing logstash-output-jdbc Installation successful #查看安裝 [admin@pe-jira gems]$ sudo /usr/share/logstash/bin/logstash-plugin list|grep jdbc logstash-filter-jdbc_static logstash-filter-jdbc_streaming logstash-input-jdbc logstash-output-jdbc
安裝數據庫的JDBC驅動
https://dev.mysql.com/downloads/connector/j/ 上傳到服務器。驅動的路徑必須嚴格一致，否則連接數據庫會報錯。
我一般都是這里下載
https://mvnrepository.com/artifact/mysql/mysql-connector-java
sudo mkdir -p /usr/share/logstash/vendor/jar/jdbc cd /usr/share/logstash/vendor/jar/jdbc sudo wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.16/mysql-connector-java-8.0.16.jar ll sudo chown -R logstash: /usr/share/logstash/vendor/jar/
配置MySQL權限
create database elk character set utf8 collate utf8_bin; grant all privileges on elk.* to elk@"%" identified by '123456'; flush privileges;
樣例-存儲Nginx訪問日志
建表
我們在數據庫中存儲數據的時候，沒有必要存儲日志的所有內容，只需存儲我們需要的重要信息即可，可以根據自身的需求進行取舍。
create table kibana_log(host varchar(128),client_ip varchar(128),url varchar(512),status int(4),responsetime float(8,3),http_user_agent varchar(512),time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin; #注：time的默認值設置為CURRENT_TIMESTAMP
mysql> desc kibana_log;+-----------------+--------------+------+-----+-------------------+-------+ | Field | Type | Null | Key | Default | Extra | +-----------------+--------------+------+-----+-------------------+-------+ | host | varchar(128) | YES | | NULL | | | client_ip | varchar(128) | YES | | NULL | | | url | varchar(512) | YES | | NULL | | | status | int(4) | YES | | NULL | | | responsetime | float(8,3) | YES | | NULL | | | http_user_agent | varchar(512) | YES | | NULL | | | time | timestamp | NO | | CURRENT_TIMESTAMP | | +-----------------+--------------+------+-----+-------------------+-------+ 7 rows in set (0.00 sec) mysql>
配置logstash.conf文件
#[admin@pe-jira conf.d]$ cat kibana.conf
input {
file {
type => "pe-jira-kibana"
path => "/home/admin/webserver/logs/kibana.log"
start_position => "beginning"
stat_interval => "2"
}
}
filter{
json {
source => "message"
skip_on_invalid_json => true
}
}
output {
if [type] == "pe-jira-kibana" {
elasticsearch {
hosts => ["10.6.76.27:9200"]
index => "logstash-pe-jira-nginx-kibana-%{+YYYY.MM.dd}"
}
jdbc {
connection_string => "jdbc:mysql://10.6.76.28/elk?user=elk&password=123456&useUnicode=true&characterEncoding=UTF8"
statement => ["insert into kibana_log(host,client_ip,url,status,responsetime,http_user_agent) VALUES(?,?,?,?,?,?)","host","clientip","url","status","responsetime","http_user_agent"] }
}
}
#注意表和日志字段一一對應
測試文件,查看是否正確:
[admin@pe-jira conf.d]$ sudo /usr/share/logstash/bin/logstash -f kibana.conf -t Thread.exclusive is deprecated, use Thread::Mutex WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2019-07-15 15:45:18.839 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [INFO ] 2019-07-15 15:45:28.447 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash [admin@pe-jira conf.d]$
刷新產生日志，看能否寫入數據庫
