ELK環境配置+log4j日志記錄

ELK環境配置+log4j日志記錄

1. 背景介紹

在大數據時代，日志記錄和管理變得尤為重要。

以往的文件記錄日志的形式，既查詢起來又不方便，又造成日志在服務器上分散存儲，管理起來相當麻煩，

想根據一個關鍵字查詢日志中某個關鍵信息相當困難。

這個時候，ELK誕生了。

什么是ELK?

簡單來說：它是一套完整的日志記錄和分析的解決方案平台。

 

2. 技術棧介紹

ELK = Elasticsearch + Logstash + Kibana

2-1) Elasticsearch：

(

Elasticsearch is a distributed open source search engine based on Apache Lucene,

and released under an Apache 2.0 license (which means that it can be downloaded, used, and modi ed free of charge).

It provides horizontal scalability, reliability, and multitenant capability for real-time search.

Elasticsearch features are available through JSON over a RESTful API.

The searching capabilities are backed by a schema-less Apache Lucene Engine,

which allows it to dynamically index data without knowing the structure beforehand.

Elasticsearch is able to achieve fast search responses because it uses indexing to search over the texts. 

)

Elasticsearch是一個分布式的開源的基於Apache Lucene項目的搜索引擎，它發布在Apache 2.0協議下

（這也就意味着它可以被免費地下載，使用並且修改）。

 Elasticsearch提供了水平的，可擴展的，可靠的，多用戶形式的實時搜索。

Elasticsearch的功能可以通過JSON格式的RESTful API形式訪問。

Elasticsearch的搜索能力是得到Apache Lucene引擎的支持，允許給文本數據增加加動態索引。

 2-2) Logstash

(

Logstash is a data pipeline that helps collect, parse, and analyze a large variety of structured and unstructured data and events generated across various systems.

It provides plugins to connect to various types of input sources and platforms, and is designed to ef ciently process logs, events,

and unstructured data sources for distribution into a variety of outputs with the use of its output plugins,

namely le, stdout (as output on console running Logstash), or Elasticsearch. 

)

Logstash是一個數據管道，它被用來收集，解析和分析各種結構化的和非結構化的由各種系統產生的數據以及事件。

它提供了插件用來連接到各種輸入數據源，可以高效地處理日志，事件以及非結構化的數據，而且可以通過輸出插件的形式

把結果輸出到各種輸出源，比如：標准輸出，控制台或者Elasticsearch。

 2-3) Kibana

(Kibana is an open source Apache 2.0 licensed data visualization platform that helps in

visualizing any kind of structured and unstructured data stored in Elasticsearch indexes.

Kibana is entirely written in HTML and JavaScript. )

Kibana是一個基於Apache 2.0協議的開源可視化平台，它用來可視化任何結構化的和非結構化的存儲在Elasticsearch索引

中的數據。Kibana完全用HTML和Javascript編寫。

 

3. 下載/安裝/配置/啟動

前面說了很多廢話，接下來是每個程序員感興趣的動手環節了，let's start!

3-1) 安裝Elasticsearch

下載地址:

https://www.elastic.co/downloads/elasticsearch

a) 解壓安裝包elasticsearch-5.2.2.tar.gz

    (由於本人是mac系統)我把它解壓到了/usr/local目錄下，完整路徑如下:

    /usr/local/elasticsearch-5.2.2

b) 編輯配置文件

   cd config

   vi elasticsearch.yml

   內容如下:

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /tmp/elasticsearch/data
#
# Path to log files:
#
path.logs: /tmp/elasticsearch/logs
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: localhost
#
# Set a custom port for HTTP:
#
http.port: 9200

c) 啟動Elasticsearch
./bin/elasticsearch

如果沒有錯誤發生，可以進入到下一步的驗證環節

d) 驗證

可以使用cURL命令:

curl 'http://localhost:9200/?pretty'

或者

直接用瀏覽器打開

http://localhost:9200

 

3-2) 安裝Logstash

a) 解壓安裝包logstash-5.2.2.tar.gz

    完整路徑如下:

    /usr/local/logstash-5.2.2

b) 編輯配置文件

   cd config

   新建配置文件log4j_es.conf

   vi log4j_es.conf

   內容如下:

input {
    log4j {
        host => "127.0.0.1"
        port => 4560
    }
}

output {
    stdout {
      codec => rubydebug
    }
    elasticsearch{
        hosts => ["localhost:9200"]
        index => "log4j-%{+YYYY.MM.dd}"
        document_type => "log4j_type"
    }
}

c) 啟動Logstash

./bin/logstash -f config/log4j-es.conf

 

3-3) 安裝Kibana

a) 解壓安裝包kibana-5.2.2.tar.gz
完整路徑如下:
/usr/local/ kibana-5.2.2

b) 編輯配置文件
cd config
vi kibana.yml
內容如下:

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

server.host: "localhost"

# The URL of the Elasticsearch instance to use for all your queries.
elasticsearch.url: "http://localhost:9200"

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
kibana.index: ".kibana"

c) 啟動Kibana

./bin/kibana

 

d) 驗證
打開網址:
http://localhost:5601/

見到如下Logo

4. Log4j記錄日志到Logstash

4-1) 新建maven項目

pom中的關鍵dependency配置如下：

<dependency>
    <groupId>log4j</groupId>
    <artifactId>log4j</artifactId>
    <version>1.2.17</version>
</dependency

 4-2) log4j.properties(放在resources文件夾下)

### 設置###
log4j.rootLogger = debug,stdout,D,E,logstash

### 輸出信息到控制抬 ###
log4j.appender.stdout = org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target = System.out
log4j.appender.stdout.layout = org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern = [%-5p] %d{yyyy-MM-dd HH:mm:ss,SSS} method:%l%n%m%n

### 輸出DEBUG 級別以上的日志到=/Users/bee/Documents/elk/log4j/debug.log###
log4j.appender.D = org.apache.log4j.DailyRollingFileAppender
log4j.appender.D.File = /Users/KG/Documents/logs/elk/debug.log
log4j.appender.D.Append = true
log4j.appender.D.Threshold = DEBUG
log4j.appender.D.layout = org.apache.log4j.PatternLayout
log4j.appender.D.layout.ConversionPattern = %-d{yyyy-MM-dd HH:mm:ss}  [ %t:%r ] - [ %p ]  %m%n

### 輸出ERROR 級別以上的日志到=/Users/bee/Documents/elk/log4j/error.log  ###
log4j.appender.E = org.apache.log4j.DailyRollingFileAppender
log4j.appender.E.File =/Users/KG/Documents/logs/elk/error.log
log4j.appender.E.Append = true
log4j.appender.E.Threshold = ERROR
log4j.appender.E.layout = org.apache.log4j.PatternLayout
log4j.appender.E.layout.ConversionPattern = %-d{yyyy-MM-dd HH:mm:ss}  [ %t:%r ] - [ %p ]  %m%n

#輸出日志到logstash
log4j.appender.logstash=org.apache.log4j.net.SocketAppender
log4j.appender.logstash.RemoteHost=127.0.0.1
log4j.appender.logstash.port=4560
log4j.appender.logstash.ReconnectionDelay=60000
log4j.appender.logstash.LocationInfo=true

4-3) Java代碼(ElkLog4jTest.java)：

package org.genesis.arena.elk;

import org.apache.log4j.Logger;

/**
 * Created by KG on 17/3/27.
 */
public class ElkLog4jTest {
    private static final Logger logger = Logger.getLogger(ElkLog4jTest.class);
    public static void main(String[] args) throws Exception {
        logger.debug("This is a debug message!");
        logger.info("This is info message!");
        logger.warn("This is a warn message!");
        logger.error("This is error message!");

        try{
            System.out.println(5/0);
        }catch(Exception e){
            logger.error(e);
        }
    }
}

4-4) 運行結果

[DEBUG] 2017-03-29 12:56:00,454 method:org.genesis.arena.elk.ElkLog4jTest.main(ElkLog4jTest.java:11)
This is a debug message!
[INFO ] 2017-03-29 12:56:00,529 method:org.genesis.arena.elk.ElkLog4jTest.main(ElkLog4jTest.java:12)
This is info message!
[WARN ] 2017-03-29 12:56:00,531 method:org.genesis.arena.elk.ElkLog4jTest.main(ElkLog4jTest.java:13)
This is a warn message!
[ERROR] 2017-03-29 12:56:00,533 method:org.genesis.arena.elk.ElkLog4jTest.main(ElkLog4jTest.java:14)
This is error message!
[ERROR] 2017-03-29 12:56:00,538 method:org.genesis.arena.elk.ElkLog4jTest.main(ElkLog4jTest.java:19)
java.lang.ArithmeticException: / by zero

然后會在Logstash控制台看到輸出如下圖:

5. 連通Kibana

5-1) 打開 http://localhost:5601/

5-2) 創建索引

   還記得我們之前在logstash配置文件中配置的索引嗎？

   log4j-%{+YYYY.MM.dd}

  因此，我們應該創建索引為：log4j-*

5-3）驗證

從這里可以看到自己剛才在Java代碼中記錄的日志。