Logstash日志寫入Mysql數據庫

一 Mysql數據庫配置

  1.1.1 mysql建庫授權

mysql> create database elk  character set utf8 collate utf8_bin;
Query OK, 1 row affected (0.00 sec)

mysql>  grant all privileges on elk.* to elk@"192.168.10.%" identified by '123456';
Query OK, 0 rows affected (0.00 sec)

mysql> flush  privileges;

    1.1.2 建表

   創建表名為：elklog

  按照自己所需在日志里面取key創建自定義的表

  time默認值沒有定義為CURRENT_TIMESTAMP的狀態

二 安裝插件

2.1.1 logstash配置mysql-connector-java包

    

   MySQL Connector/J是MySQL官方JDBC驅動程序，JDBC（Java Data Base Connectivity,java數據庫連接）是一種用於執行SQL語句的Java API，可以為多種關系數據庫提供統一訪問，它由一組用Java語言編寫的類和接口組成。

  官方下載地址：https://dev.mysql.com/downloads/connector/

  下載地址：https://dev.mysql.com/downloads/connector/j/

mkdir -p /usr/share/logstash/vendor/jar/jdbc
cd /usr/share/logstash/vendor/jar/jdbc
rz
ls
tar xf mysql-connector-java-8.0.12.tar.gz 
mv mysql-connector-java-8.0.12/mysql-connector-java-8.0.12.jar .
mv mysql-connector-java-8.0.12 mysql-connector-java-8.0.12.tar.gz /tmp
[root@localhost vendor]# pwd
//usr/share/logstash/vendor
[root@localhost vendor]# chown -R logstash.logstash jar

 1.1.2 安裝 logstash-output-jdbc插件

    更改gem源：

       國外的gem源由於網絡原因，從國內訪問太慢而且不穩定，還經常安裝不成功，因此之前一段時間很多人都是使用國內淘寶的gem源https://ruby.taobao.org/，現在淘寶的gem源雖然還可以使用已經停止維護更新，其官方介紹推薦使用https://gems.ruby-china.org。

yum install gem
gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org/ 
[root@localhost ~]# gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org/ 
source https://gems.ruby-china.org/ already present in the cache
source https://rubygems.org/ not present in cache
[root@localhost ~]#  gem source list
*** CURRENT SOURCES ***

https://gems.ruby-china.org/

    安裝插件：

[root@localhost ~]# /usr/share/logstash/bin/logstash-plugin   install  
[root@localhost ~]# /usr/share/logstash/bin/logstash-plugin  list

三 配置logstash Nginx配置文件

[root@localhost ~]# cat /etc/logstash/conf.d/nginx.conf 
input {
      file {
          path => "/opt/vhosts/fatai/logs/access_json.log"
              start_position => "beginning"
          type => "nginx-accesslog"
          codec => json
                  stat_interval => "2"          
      }
      file {
           path => "/var/log/messages" 
               start_position => "beginning" 
           type => "systemlog-test"   
                   stat_interval => "2"          
      }
      file {
           type => "nginx-error" 
           path => [ "/var/log/nginx/error.log" ]
           tags => [ "nginx","error"]
                start_position => "beginning"
                   stat_interval => "2"          
       }
}


output {
    if [type] == "nginx-accesslog" {
          elasticsearch {
          hosts => ["192.168.10.10:9200"]
              index => "logstash-nginx-access-test-%{+YYYY.MM.dd}"
      }
 jdbc { connection_string => "jdbc:mysql://192.168.10.172/elk?user=elk&password=123456&useUnicode=true&characterEncoding=UTF8" statement => ["INSERT INTO elklog(host,clientip,url,responsetime,upstreamtime) VALUES(?,?,?,?,?)", "host","clientip","url","responsetime","upstreamtime"]} } if [type] == "systemlog-test" {
      elasticsearch {
           hosts => ["192.168.10.10:9200"]
           index => "logstash-systemlog-test-%{+YYYY.MM.dd}"
      }}
    if [type] == "nginx-error" {
      elasticsearch {
          hosts => ["192.168.10.10:9200"]
          index => "logstash-nginx-error-test-%{+YYYY.MM.dd}"                                                                      
    }}
}

   驗證配置文件並重啟Logstash

[root@localhost ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@localhost ~]# systemctl restart logstash.service

四 數據庫查看數據