(原)logstash-forwarder + logstash + elasticsearch + kibana

[logstash-forwarder + logstash + elasticsearch + kibana]
------------------------------------------------------------------------------------------------------------------------------------------------
摘要:logstash-forwarder搜集日志，匯總給logstash，然后輸出到elasticsearch,並由kibana展現web界面.
------------------------------------------------------------------------------------------------------------------------------------------------
一 安裝
1.logstash-forwarder
see and install:
https://github.com/elasticsearch/logstash-forwarder

(logstash-forwarder有個坑. 雖然嚴格講不算是logstash-forwarder的坑.
跟證書相關的:https://github.com/elasticsearch/logstash-forwarder/issues/221 <-可以不看.
下面的解決方案規避這個坑了. 下面會提到.)

2.logstash
see and install:
http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash

3.elasticsearch

3.1.下載https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.2.tar.gz

3.2.解壓到目錄 elasticsearch-1.3.2

3.3. 測試安裝是否成功
$ cd elasticsearch-1.3.2/
$ bin/elasticsearch
$ curl -X GET http://localhost:9200/
(保持elasticsearch一直運行. 下面將繼續測試)

4.kibana:

4.1.下載https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz

4.2. 解壓到目錄 kibana-3.1.0

4.3. 測試安裝是否成功
$ cd kibana-3.1.0
$ vi config.js
第32行修改為：
elasticsearch: "http://localhost:9200",
或者如果是要非本地訪問,就應該這樣:
elasticsearch: "http://"+window.location.hostname+":9200"
注意后面有逗號.
在瀏覽器里打開這目錄里的index.html.

------------------------------------------------------------------------------------------------------------------------------------------------
二 .方案:

client[logstash-forwarder]---|
client[logstash-forwarder]---|---log-server[logstash]--->[elasticsearch]
client[logstash-forwarder]---|

2.1 先啟動elasticsearch
前面已經啟動了.

2.2  開啟logstash
先寫logstash的配置文件:
$ cd logstash-1.4.2
$ vi test_logstash.conf
input {
  lumberjack {
    # The port to listen on
    port => 5000

    # The paths to your ssl cert and key
    ssl_certificate => "/home/xiaou/logstash-forwarder.crt"
    ssl_key => "/home/xiaou/logstash-forwarder.key"

    # Set this to whatever you want.
    type => "somelogsXXX"
  }
}
output {
  elasticsearch { host => localhost } # 因為logstash和elasticsearch在同一台機器上，所以這里可以用localhost
  stdout { codec => rubydebug }
}

還要產生自簽證書:
$ openssl req -subj '/CN=localhost/' -x509 -batch -nodes -newkey rsa:2048 -keyout /home/xiaou/logstash-forwarder.key -out /home/xiaou/logstash-forwarder.crt  -days 1095
(這里用“-subj '/CN=localhost/'”規避了上面提到的logstash-forwarder的坑)

然后啟動logstash：
$ bin/logstash -f test_logstash.conf

2.3 啟動logstash-forwarder
先寫logstash-forwarder的配置文件:
$ cd logstash-forwarder
$ vi test_forwarder.conf
{
  "network": {
    "servers": [ "localhost:5000" ],
 "ssl ca": "/home/xiaou/logstash-forwarder.crt",
    "timeout": 5
  },
  "files": [
    {
      "paths": [
        "/var/log/linshi.txt",
        "/var/log/*.log"
      ],
      "fields": {
        "type": "linshiXX"
      }
    }
  ]
}
(這里配置文件的寫法也是規避了前面提到的logstash-forwarder的坑:servers沒用ip)
啟動logstash-forwarder:
$ ./logstash-forwarder -config test_forwarder.conf
logstash-forwarder啟動后就會與logstash建立tcp連接.

測試, 寫日志，觀察運行logstash的終端的輸出:
$ echo 1234 >> /var/log/linshi.txt

2.4  打開kibana,展現最終匯總到elasticsearch的日志.
(唯kibana不能算是服務, 它只是一個“閱讀器”.)
用瀏覽器打開kibana-3.1.0目錄下的index.html,看右邊倒數第五行有個鏈接。打開.
------------------------------------------------------------------------------------------------------------------------------------------------
三.深入：

1. type
logstash.conf里的
input {
  lumberjack {
    ...
    type => "this forwarder's file have no type!"
這個type,是對forwarder.conf的補充:如果forwarder.conf里沒有type，則這里的type就會填充日志event的type字段.
ps:
一條日志event是這樣的:
{
       "message" => "xx",
      "@version" => "1",
    "@timestamp" => "2014-09-18T03:31:12.744Z",
          "type" => "linshi1",
          "file" => "/var/log/epoch/linshi.txt",
          "host" => "xiaou-mint",
        "offset" => "568"
}
用type來作為區分各個日志應該不錯:
在forwarder里這樣寫files:
  "files": [
    {
      "paths": [
        "/var/log/epoch/linshi1.txt"
      ],
      "fields": {
        "type": "linshi1"
      }
    },
    {
      "paths": [
        "/var/log/epoch/linshi2.txt"
      ],
      "fields": {
        "type": "linshi2"
      }
    }
  ]

2.add_field添加字段
    add_field => {
      "test_field" => "asdasd"
      "test_filed2" => "112233"
    }
盡量不要跟日志event里已有的字段沖突了，如果要這么做，需要自行測試是否會覆蓋event日志的字段. 我測試了幾個字段諸如type、message、file，居然表現各一,無法統一下結論.

3.if表達式
隨時需要查文檔http://logstash.net/docs/1.4.2/。。。不寫了. End.
/*
http://logstash.net/docs/1.4.2/inputs/lumberjack
http://logstash.net/docs/1.4.2/configuration#conditionals
http://logstash.net/docs/1.4.2/filters/mutate
http://logstash.net/docs/1.4.2/filters/drop
*/

4. 最后給出兩個conf的測試內容:
logstash.conf :

input {
  lumberjack {
    # The port to listen on
    port => 5000

    # The paths to your ssl cert and key
    ssl_certificate => "/home/xiaou/logstash-forwarder.crt"
    ssl_key => "/home/xiaou/logstash-forwarder.key"

    type => "this forwarder's file have no type!"

  }
}

filter{
  if [type] == "linshi2"{
    mutate{
      replace => ["message","%{message}:it's linshi2"]    
      update => ["file", "FILE_LINSHI2"] # 替換字段.
    }
  }else{ # linshi1
    if "error" in [message]{ # 日志里還有“error”字符串
      mutate{
        add_field => {"NOTE" => "ERROR!"} # 添加字段
        add_tag => "tag_error!" # 添加標簽. 標簽是個數組
        add_tag => "tag_error2!"
      }
    }else{ # 如果來自linshi1.txt的並且沒有“error”自負, 則丟棄.
      drop{}
    }
  }
}

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

forwarder.conf :

{
  "network": {
    "servers": [ "localhost:5000" ],
 "ssl ca": "/home/xiaou/logstash-forwarder.crt",
    "timeout": 5
  },

  "files": [
    {
      "paths": [ 
        "/var/log/epoch/linshi1.txt"
      ],
      "fields": { 
        "type": "linshi1" 
      }
    },
    {
      "paths": [ 
        "/var/log/epoch/linshi2.txt"
      ],
      "fields": { 
        "type": "linshi2" 
      }
    }
  ]
}

------------------------------------------------------------------------------------------------------------------------------------------------
End.