服務端安裝
Elasticsearch和Kibana（需要安裝openjdk1.8以上）
安裝方法：https://www.elastic.co
以Ubuntu為例：
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
apt-get update
apt-get install elasticsearch
apt-get install kibana
elasticsearch配置
cat /etc/elasticsearch/elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 172.16.80.28
http.port: 9200
discovery.zen.ping.unicast.hosts: ["172.16.80.28","172.16.80.155"]
附件有文件elasticsearch.map
Kibana配置
cat /etc/kibana/kibana.yml
server.port: 5601
server.host: "172.16.80.28"
elasticsearch.url: "http://172.16.80.28:9200"
Elastalert（需要Python2.7）
下載 https://github.com/Yelp/elastalert/releases
進入elastalert目錄
pip install -r requirements.txt
python setup.py install
elastalert-create-index
前台運行elastalert
python -m elastalert.elastalert --verbose --rule rules/my_rule.yaml
安裝參考:
https://elastalert.readthedocs.io/en/latest/running_elastalert.html#downloading-and-configuring
https://blog.csdn.net/df007df/article/details/54773391
elastalert配置
cat /etc/elastalert/rules/my_rule.yaml
es_host: 172.16.80.28
es_port: 9200
name: filebeat rules
type: frequency
index: filebeat*
num_events: 5
timeframe:
hours: 1
filter:
- query:
query_string:
query: "message: *error*"
alert:
- "email"
email:
- "aaa@qq.com"
- "bbb@qq.com"
alert_text: "Ref Log http://172.16.80.28:5601/app/kibana"
smtp_host: smtp.exmail.qq.com
smtp_port: 25
smtp_ssl: false
smtp_auth_file: /etc/elastalert/rules/smtp_auth_file.yaml
from_addr: aaa@qq.com
cat /etc/elastalert/rules/smtp_auth_file.yaml
user: "aaa@qq.com"
password: "郵箱密碼"
cat /etc/elastalert/config.yaml
rules_folder: /etc/elastalert/rules
run_every:
minutes: 1
buffer_time:
minutes: 15
es_host: 172.16.80.28
es_port: 9200
s_url_prefix: elasticsearch
writeback_index: elastalert_status
alert_time_limit:
days: 2
elastalert配置注解
rules_folder：用來加載下一階段rule的設置，默認是example_rules
run_every：用來設置定時向elasticsearch發送請求
buffer_time：用來設置請求里時間字段的范圍，默認是45分鍾
es_host：elasticsearch的host地址
es_port：elasticsearch 對應的端口號
use_ssl：可選的，選擇是否用SSL連接es，true或者false
es_username：es認證的username
es_password：es認證的password
writeback_index：elastalert產生的日志在elasticsearch中的創建的索引
alert_time_limit：失敗重試的時間限制
告警參考
https://blog.csdn.net/gamer_gyt/article/details/52917116
https://elastalert.readthedocs.io/en/latest/ruletypes.html
客戶端安裝
Filebeat安裝
以Ubuntu為例：
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
apt-get update
apt-get install filebeat
filebeat配置
cat /etc/filebeat/filebeat.yml
filebeat.prospectors:
- type: log
enabled: true
paths:
- /var/log/nginx/*.log
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
setup.kibana:
host: "172.16.80.28:5601"
output.elasticsearch:
hosts: ["172.16.80.28:9200"]
EFK備注
日志位置 /var/lib/elasticsearch/nodes/
日志大小 暫時為345Mb
日志輪循 暫時不知道在哪設置
elasticsearch.map內容：
# uint mapping { "settings" : { "index" : { "number_of_replicas" : 1, "number_of_shards" : 5 } }, "mappings" : { "values" : { "properties" : { "itemid" : { "type" : "long" }, "clock" : { "format" : "epoch_second", "type" : "date" }, "value" : { "type" : "long" } } } } } # dbl mapping { "settings" : { "index" : { "number_of_replicas" : 1, "number_of_shards" : 5 } }, "mappings" : { "values" : { "properties" : { "itemid" : { "type" : "long" }, "clock" : { "format" : "epoch_second", "type" : "date" }, "value" : { "type" : "double" } } } } } # str mapping { "settings" : { "index" : { "number_of_replicas" : 1, "number_of_shards" : 5 } }, "mappings" : { "values" : { "properties" : { "itemid" : { "type" : "long" }, "clock" : { "format" : "epoch_second", "type" : "date" }, "value" : { "fields" : { "analyzed" : { "index" : true, "type" : "text", "analyzer" : "standard" } }, "index" : false, "type" : "text" } } } } } # text mapping { "settings" : { "index" : { "number_of_replicas" : 1, "number_of_shards" : 5 } }, "mappings" : { "values" : { "properties" : { "itemid" : { "type" : "long" }, "clock" : { "format" : "epoch_second", "type" : "date" }, "value" : { "fields" : { "analyzed" : { "index" : true, "type" : "text", "analyzer" : "standard" } }, "index" : false, "type" : "text" } } } } } # log mapping { "settings" : { "index" : { "number_of_replicas" : 1, "number_of_shards" : 5 } }, "mappings" : { "values" : { "properties" : { "itemid" : { "type" : "long" }, "clock" : { "format" : "epoch_second", "type" : "date" }, "value" : { "fields" : { "analyzed" : { "index" : true, "type" : "text", "analyzer" : "standard" } }, "index" : false, "type" : "text" } } } } }