基於Kibana的可視化監控報警插件sentinl入門

sentinl是什么

• Kibi/Kibana Alert & Reporting App
• Watching your data, 24/7/365

sentinl是一個免費的kibana預警與報告插件，與付費軟件X-Pack功能類似。

Some Examples for illustration:

• HIT COUNT PER HOUR
• QUESTION: How many hits does index X receive hourly?
• WATCHER: query index and return count of hits in last hour
• ACTION: Notify with number of Hits per hour
• METRIC THRESHOLDS
• QUESTION: Is any of my monitored metrics surpassing a certain value?
• WATCHER: query index and type for specific values, aggregated by an arbitrary field.
• ACTION: Notify with aggs bucket details every time a threshold is surpassed or spike anomaly detected.
• BLACKLISTS HITS
• QUESTION: Is any of my users trying to reach blacklisted destinations?
• WATCHER: query firewall logs comparing destination IPs to a blacklist.
• ACTION: Notify admin via email if any IP >= 10 matches returned
• FAILED LOGINS
• QUESTION: Are there recurring failure attempts authenticating users on my network?
• WATCHER: query active directory logs for login failures in last hour and compare to user index. .
• ACTION: Notify admin via webhook if >= 10 matches returned
• LEAK DETECTION (chain)
• QUESTION: Are there any public leaks about my data I was not aware of?
• WATCHER: query for user emails included in published leaks ingested from third parties.
• ACTION: Save hits in secondary result Index. Notify via email if leak was not known in a secondary Watcher

安裝

./kibana-plugin install file:./sentinl-v6.0.1.zip

安裝完成后，要重啟kibana

fuser -n tcp 5601  
ps -ef | grep node
kill -9 pid

./kibana &

使用步驟

使用包括5個步驟

• Step 1: New Watcher

give our Watcher a name and choose an execution frequency

• Step 2: Input Query

es的搜索與聚合

• Step 3: Condition

validate if the results received back are worth processing

語法與x-pack script condition語法類似

相當於過濾條件

"condition": {  
      "script": {  
        "script": "payload.hits.total>=1"  //當報警條件為***出現的次數大於1  
      }  
    }  


"condition": {
      "script": {
        "script": "payload.hits.hits[0]._source.responsetime > 0.01" // 檢索條件 響應時間大於 0.01秒
      }
    }

• Step 4: Transform

Our data might need adjustments or post processing. Process our payload using a javascript expression/script

事后處理

• Step 5: Actions

Let's form a notification using the mustache templating language。

可以采用多種方式發送通知。

transform

How to Adapt or Post-Process data

Post Process事后的處理。

The transform script is the wild member of the family and can be used to inject simple or complex logic into the pipeline before delivery to actions using pure javascript.

From converting format types, through generating brand new payload keys and interpolating data, transform is the way up. The script expects a boolean condition to trigger actions. A false condition can be forced to stop the execution. BONUS: Transforms can be saved and used across Watchers! "transform": { "script": { "script": "payload.newvar = payload.aggs.some.values['95.0']" } }

action舉例之郵件發送

• Troubleshooting

kibana.yml

logging.verbose: true
sentinl:
  settings:
    email:
      active: true
      host: smtp.exmail.qq.com
      ssl: false
    report:
      active: true
      tmp_path: /tmp/
      

上面是官網的，下面是實踐已OK      
sentinl:
  settings:
    email:
      active: true
      user: [e-mail]
      password: Dd@2016
      host: smtp.exmail.qq.com
      ssl: true
      timeout: 10000
    report:
      active: true
      tmp_path: /tmp/
      

先測試

mailx -S smtp=<smtp-server-address> -r <from-address> -s <subject> -v <to-address> < body.txt

yum -y install sendmail
yum install -y sendmail-cf

/etc/init.d/sendmail start
chkconfig sendmail on
yum install -y mailx

vim /etc/mail.rc（optional）

set from=[e-mail]
set smtp=smtp.exmail.qq.com
set smtp-auth-user=[e-mail]
set smtp-auth-password=******
set smtp-auth=login
set nss-config-dir="/etc/pki/nssdb/"

test

echo "This is the message body and contains the message" | mailx -v -r "[e-mail]" -s "This is the subject" -S smtp="smtp.exmail.qq.com" -S smtp-use-starttls -S smtp-auth=login -S smtp-auth-user="[e-mail]" -S smtp-auth-password="******" -S ssl-verify=ignore -S nss-config-dir="/etc/pki/nssdb/" [e-mail]

參考文獻

• 基於Kibana的可視化監控報警插件 KAAE 的配置
• http://sentinl.readthedocs.io/en/latest/
• Kibana 可視化監控報警插件 KAAE 的介紹與使用