ELK監控系統nginx / mysql慢日志
ELK監控系統nginx日志
1.環境准備
centos6.8_64 mini
IP:192.168.10.78
tar包：
logstash-2.4.0.tar.gz
elasticsearch-2.4.0.tar.gz
kibana-4.6.1-linux-x86_64.tar.gz
JDK環境
| elasticsearch | logstach | kibana | java |
|---|---|---|---|
| V2.4 | V2.4 | V4.6.1 | V1.8.0_111 |
nginx采用yum安裝即可，安裝后啟動此服務。
2.logstash簡單配置編寫
logstash目錄下新建測試文件test.conf 其內動如下
- input {
- file {
- type => "syslog"
- tags => ["log"]
- path => ["/var/log/messages","/log/*.log"]
- start_position => beginning
- ignore_older => 0
- }
- file {
- type => "nginx_log"
- tags => ["nginx"]
- path => ["/var/log/nginx/access.log"]
- start_position => beginning
- ignore_older => 0
- }
- }
-
-
- output
- {
-
- elasticsearch {
- hosts => [ "192.168.10.78:9200" ]
- }
-
- }
編寫好后測試運行：
- [root@localhost logstash]# ./bin/logstash -f test.conf
- Settings: Default pipeline workers: 2
- Pipeline main started
此時logstash已經運行，我們通過瀏覽器訪問web：192.168.10.68
同時在終端下重啟網卡服務，用以查看系統日志是否有輸出至kibana
打開kibana查看日志輸出
可以看見有網卡日志和web訪問日志。
接下來監控mysql日志
yum install mysql mysql-server -y
安裝完直接在kibana就看見安裝mysql的日志輸出結果了
Time type tags message
November 10th 2016, 10:18:13.193 syslog log Nov 10 10:18:12 localhost yum[11445]: Installed: mysql-server-5.1.73-7.el6.x86_64
November 10th 2016, 10:18:13.057 syslog log Nov 10 10:18:12 localhost yum[11445]: Installed: mysql-server-5.1.73-7.el6.x86_64
November 10th 2016, 10:18:10.187 syslog log Nov 10 10:18:09 localhost yum[11445]: Installed: mysql-5.1.73-7.el6.x86_64
November 10th 2016, 10:18:10.185 syslog log Nov 10 10:18:09 localhost yum[11445]: Installed: perl-DBD-MySQL-4.013-3.el6.x86_64
November 10th 2016, 10:18:10.183 syslog log Nov 10 10:18:09 localhost yum[11445]: Installed: perl-DBI-1.609-4.el6.x86_64
November 10th 2016, 10:18:10.051 syslog log Nov 10 10:18:09 localhost yum[11445]: Installed: mysql-5.1.73-7.el6.x86_64
November 10th 2016, 10:18:10.050 syslog log Nov 10 10:18:09 localhost yum[11445]: Installed: perl-DBD-MySQL-4.013-3.el6.x86_64
November 10th 2016, 10:18:10.048 syslog log Nov 10 10:18:09 localhost yum[11445]: Installed: perl-DBI-1.609-4.el6.x86_64
接下來修改test.conf配置文件，添加對mysql滿日志的監控
1.yum install mysql mysql-server
2.service network restart
3.mysql慢日志開啟:
(此處參考http://blog.itpub.net/29500582/viewspace-1432985/)
Mysql 啟動慢查詢日志 （不用重啟）
查看mysql系統參數
mysql> show variables like "%slow%";
+---------------------------+-------------------------------+
| Variable_name | Value |
+---------------------------+-------------------------------+
| log_slow_admin_statements | OFF |
| log_slow_slave_statements | OFF |
| slow_launch_time | 2 |
| slow_query_log | OFF |
| slow_query_log_file | /mysql/data/nagiosdb-slow.log |
+---------------------------+-------------------------------+
5 rows in set (0.00 sec)
slow_query_log： off關閉狀態 on開啟狀態
slow_launch_time 默認超過2s為慢查詢
slow_query_log_file 慢查詢日志存放地點
這三個參數，在不同的mysql版本中，不太一樣，不過都可以通過 show variables like "%slow%" 查看出來
運行如下命令即可運行慢查詢日志
mysql> set global slow_query_log=ON;
Query OK, 0 rows affected (0.03 sec)
mysql> set global slow_launch_time=5;
Query OK, 0 rows affected (0.00 sec)
mysql> show variables like "%slow%";
+---------------------------+-------------------------------+
| Variable_name | Value |
+---------------------------+-------------------------------+
| log_slow_admin_statements | OFF |
| log_slow_slave_statements | OFF |
| slow_launch_time | 5 |
| slow_query_log | ON |
| slow_query_log_file | /mysql/data/nagiosdb-slow.log |
+---------------------------+-------------------------------+
5 rows in set (0.00 sec)
mysql 5.1.6版本起，slow_query_log 和 slow_launch_time 支持寫文件或寫數據庫表兩種方式，並且日志的開啟，輸出方式的修改，都可以在global級別動態修改。
只需簡單通過set global slow_query_log=ON;即可開啟慢查詢，而不需要重啟數據庫！
可以直接寫到配置文件中 my.cnf
slow_query_log_file=/mysql/log/nagiosdb-slow.log
slow_launch_time=5
可以完成配置！！
！！！根據上述配置開啟慢日志后，查詢本機日志名稱及目錄，不要搞錯。
在test.conf配置文件添加如下：
- # this is mysql log
- file {
- type => "mysql_log"
- tags => ["mysql"]
- path => ["/var/log/mysql/mysqld.log"]
- start_position => beginning
- ignore_older => 0
- }
- # this is mysql-slow log
- file {
- type => "mysql_slow"
- tags => ["mysql-slow"]
- path => ["/var/run/mysqld/mysqld-slow.log"]
- start_position => beginning
- ignore_older => 0
- }
配置完后啟動logstash
[root@localhost logstash]# ./bin/logstash -f test.conf
確認慢查詢已開啟
mysql> show variables like "%slow%";
+---------------------+---------------------------------+
| Variable_name | Value |
+---------------------+---------------------------------+
| log_slow_queries | ON |
| slow_launch_time | 2 |
| slow_query_log | ON |
| slow_query_log_file | /var/run/mysqld/mysqld-slow.log |
+---------------------+---------------------------------+
4 rows in set (0.00 sec)
mysql>
輸入測試命令，並查看kibana是否有mysql-low數據輸出；
mysql> select sleep(6);
+----------+
| sleep(6) |
+----------+
| 0 |
+----------+
1 row in set (5.99 sec)
mysql>
慢數據輸出正常：
!!!目前所有日志數據均未作規范化輸出處理，只是簡單測試了elk的基本功能，后續將升入研究學習elk.后面會繼續做好筆記供自己參考學習。