碼上快樂

相關內容    簡體    繁體

ELK實踐（二）：收集Nginx日志

本文轉載自   查看原文   2018-12-22 13:33   1474    Elasticsearch/ ELK

Nginx訪問日志

這里補充下Nginx訪問日志使用的說明。一般在nginx.conf主配置文件里需要定義一種格式：

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    '$status $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for" $request_time';

上面的格式我是基於默認的加了一個$request_time。

然后子配置使用：

access_log  logs/myapp.log  main;

即可。

Filebeat采集日志數據到ElasticSearch

配置：

su -e elk
cd /usr/local/elk
vim beats/filebeat/filebeat.test_nginx.yml

配置詳情：

filebeat.prospectors:
- type: log
  input_type: log
  paths:
    - /work/yphp/nginx/logs/*.log
  tags: ["ngx", "yujc"]
  fields:
    logIndex: nginx
    docType: nginx-access
  fields_under_root: true
  tail_files: false

output.elasticsearch:
  hosts: ["127.0.0.1:9200"]
  index: "test-nginx-%{+yyyy.MM.dd}"

配置說明：

filebeat.prospectors:

  • type 日志類型，默認log
  • input_type 輸入類型，默認log
  • paths 采集的日志，可以使用通配符。支持多個
  • tags 自定義標簽，是個數組。自定義
  • fields 自定義字段
  • fields_under_root 自定義字段是否追加到根。如果為false，fields配置的字段鍵名是fields
  • tail_files 是否從末尾開始采集
  • document_type 自定義字段，用於Logsatsh區分來源，在Logsatsh里用變量type表示

output.elasticsearch:

  • hosts 配置ES節點，數組格式，支持多個。
  • index 配置ES索引。不配置使用默認的 filebeat-*
  • protocol 配置協議，例如http,https
  • username 配置ES用戶名，例如elastic
  • password 配置ES密碼，例如changeme

設置權限600，並啟動filebeat:

chmod -R 600 beats/filebeat/filebeat.test_nginx.yml

./beats/filebeat/filebeat -c beats/filebeat/filebeat.test_nginx.yml

然后訪問Nginx應用，查看ES是否新增了一個索引：


$ curl http://127.0.0.1:9200/_cat/indices?v | grep test-nginx
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
105  1161  105  1161    0     0   123k      0 --:--:-- --:--:-- --:--:--  125k
yellow open   test-nginx-2018.09.24             ArxrVVOkTjG8ZlXJjb9bVg   5   1          1            0     11.6kb         11.6kb

我們查看一條數據：

$ curl http://127.0.0.1:9200/test-nginx-2018.09.24/_search?q=*&size=1


{
  "_index": "test-nginx-2018.09.24",
  "_type": "doc",
  "_id": "AWYKkBqtJzfnbYlB_DRX",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2018-09-24T07:51:43.140Z",
    "beat": {
      "hostname": "2106567e5bce",
      "name": "2106567e5bce",
      "version": "5.6.2"
    },
    "docType": "nginx-access",
    "input_type": "log",
    "logIndex": "nginx",
    "message": "172.16.10.1 - - [24/Sep/2018:07:51:40 +0000] \"GET /?time=22 HTTP/1.1\" 200 97991 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36\" \"-\" 0.009",
    "offset": 5243,
    "source": "/work/yphp/nginx/logs/hello71.log",
    "tags": [
      "ngx",
      "yujc"
    ],
    "type": "log"
  },
  "fields": {
    "@timestamp": [
      1537775503140
    ]
  },
  "sort": [
    1537775503140
  ]
}

可以看到已經有數據了。但是日志內容作為一個整體（字段是message）了。

Filebeat采集日志數據，Logstash過濾發到ElasticSearch

配置：

su -e elk
cd /usr/local/elk
vim beats/filebeat/filebeat.test_nginx2.yml

配置詳情：

filebeat.prospectors:
- type: log
  input_type: log
  paths:
    - /work/yphp/nginx/logs/*.log
  tags: ["ngx", "yujc"]
  fields:
    logIndex: nginx
    docType: nginx-access
  fields_under_root: true
  tail_files: false

output.logstash:
  hosts: ["127.0.0.1:5044"]

配置logstash

su -e elk
cd /usr/local/elk
vim logstash/config/conf.d/filebeat.conf

配置詳情：

input {
    beats {
        port => 5044
    }
}

filter {
    grok {
        match => { "message" => "%{IPORHOST:remote_ip} - %{DATA:user_name} \[%{HTTPDATE:time}\] \"%{WORD:method} %{DATA:url} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:body_sent:bytes} \"%{DATA:referrer}\" \"%{DATA:agent}\" \"%{DATA:x_forwarded_for}\" %{NUMBER:request_time}" }
        remove_field => "message"
    }
}

output {
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
        index => "test-nginx2-%{type}-%{+YYYY.MM.dd}"
        document_type => "%{type}"
    }
    stdout { codec => rubydebug }
}

我使用的nginx日志格式是在標准格式后面加了2個字段$http_x_forwarded_for和$request_time：

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" ' 
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" $request_time';

日志示例：

172.16.10.1 - - [24/Sep/2018:09:04:40 +0000] "GET /?time=2244 HTTP/1.1" 200 98086 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36" "-" 0.002

上面的grok表達式是：

%{IPORHOST:remote_ip} - %{DATA:user_name} \[%{HTTPDATE:time}\] \"%{WORD:method} %{DATA:url} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:body_sent:bytes} \"%{DATA:referrer}\" \"%{DATA:agent}\" \"%{DATA:x_forwarded_for}\" %{NUMBER:request_time}

我們先使用Grok Debugger 工具在線調試下，看看寫的grok是否正確。我之前沒有測試之前啟動，發現ES里沒有grok里解析出來的字段，后來在命令行看到filebeat的輸出（前台運行）：

$ ./beats/filebeat/filebeat -c beats/filebeat/filebeat.test_nginx2.yml

{
    "@timestamp" => 2018-09-24T09:01:19.555Z,
      "logIndex" => "nginx",
        "offset" => 6467,
       "docType" => "nginx-access",
      "@version" => "1",
    "input_type" => "log",
          "beat" => {
            "name" => "2106567e5bce",
        "hostname" => "2106567e5bce",
         "version" => "5.6.2"
    },
          "host" => "2106567e5bce",
        "source" => "/work/yphp/nginx/logs/hello71.log",
       "message" => "172.16.10.1 - - [24/Sep/2018:09:01:14 +0000] \"GET /?time=2244 HTTP/1.1\" 200 98087 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36\" \"-\" 0.195",
          "type" => "log",
          "tags" => [
        [0] "ngx",
        [1] "yujc",
        [2] "beats_input_codec_plain_applied",
        [3] "_grokparsefailure"
    ]
}

最后面提示了_grokparsefailure，說明grok部分寫的有問題。由於是參考的網上教程，也加上剛接觸，不知道怎么配置，filebeat.conf調試了很久才生效。

我們打開Grok Debugger，第一個輸入框輸入filebeat采集的消息原文message字段里的內容，第二個輸入框輸入grok表達式：

點擊Go按鈕即可解析。如果下面的內容是{}說明解析失敗，然后可以修改表達式，該工具會自動解析。最終解析結果：

{
  "remote_ip": [
    [
      "172.16.10.1"
    ]
  ],
  "HOSTNAME": [
    [
      "172.16.10.1"
    ]
  ],
  "IP": [
    [
      null
    ]
  ],
  "IPV6": [
    [
      null
    ]
  ],
  "IPV4": [
    [
      null
    ]
  ],
  "user_name": [
    [
      "-"
    ]
  ],
  "time": [
    [
      "24/Sep/2018:08:47:59 +0000"
    ]
  ],
  "MONTHDAY": [
    [
      "24"
    ]
  ],
  "MONTH": [
    [
      "Sep"
    ]
  ],
  "YEAR": [
    [
      "2018"
    ]
  ],
  "TIME": [
    [
      "08:47:59"
    ]
  ],
  "HOUR": [
    [
      "08"
    ]
  ],
  "MINUTE": [
    [
      "47"
    ]
  ],
  "SECOND": [
    [
      "59"
    ]
  ],
  "INT": [
    [
      "+0000"
    ]
  ],
  "method": [
    [
      "GET"
    ]
  ],
  "url": [
    [
      "/?time=2244"
    ]
  ],
  "http_version": [
    [
      "1.1"
    ]
  ],
  "BASE10NUM": [
    [
      "1.1",
      "200",
      "98086",
      "0.002"
    ]
  ],
  "response_code": [
    [
      "200"
    ]
  ],
  "body_sent": [
    [
      "98086"
    ]
  ],
  "referrer": [
    [
      "-"
    ]
  ],
  "agent": [
    [
      "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
    ]
  ],
  "x_forwarded_for": [
    [
      "-"
    ]
  ],
  "request_time": [
    [
      "0.002"
    ]
  ]
}

然后可以啟動logstash了。

測試logstash配置是否通過：

./logstash/bin/logstash -f logstash/config/conf.d/filebeat.conf --config.test_and_exit

Config Validation Result: OK. Exiting Logstash

# 啟動logstash
./logstash/bin/logstash &

# 啟動filebeat
./beats/filebeat/filebeat -c beats/filebeat/filebeat.test_nginx2.yml

我們再次訪問Nginx應用，然后我們查看一條數據：

$ curl http://127.0.0.1:9200/test-nginx2-log-2018.09.24/_search?q=*&size=1&sort=@timestamp:desc

{
  "took": 14,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 3,
    "max_score": null,
    "hits": [
      {
        "_index": "test-nginx2-log-2018.09.24",
        "_type": "log",
        "_id": "AWYK0to8JzfnbYlB_DRx",
        "_score": null,
        "_source": {
          "response_code": "200",
          "agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36",
          "logIndex": "nginx",
          "offset": 6875,
          "method": "GET",
          "docType": "nginx-access",
          "user_name": "-",
          "input_type": "log",
          "http_version": "1.1",
          "source": "/work/yphp/nginx/logs/hello71.log",
          "message": """172.16.10.1 - - [24/Sep/2018:09:04:40 +0000] "GET /?time=2244 HTTP/1.1" 200 98086 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36" "-" 0.002""",
          "type": "log",
          "url": "/?time=2244",
          "tags": [
            "ngx",
            "yujc",
            "beats_input_codec_plain_applied"
          ],
          "x_forwarded_for": "-",
          "referrer": "-",
          "@timestamp": "2018-09-24T09:04:40.404Z",
          "remote_ip": "172.16.10.1",
          "request_time": "0.002",
          "@version": "1",
          "beat": {
            "name": "2106567e5bce",
            "hostname": "2106567e5bce",
            "version": "5.6.2"
          },
          "host": "2106567e5bce",
          "body_sent": "98086",
          "time": "24/Sep/2018:09:04:40 +0000"
        },
        "sort": [
          1537779880404
        ]
      }
    ]
  }
}

里面就包含了所有我們解析出來的字段。

kibana里查看

打開kibana web地址：http://127.0.0.1:5601，依次打開：Management
-> Kibana -> Index Patterns ,選擇Create Index Pattern：
a. Index pattern 輸入：test-nginx2-* ；
b. Time Filter field name 選擇 @timestamp。
c. 點擊Create。

然后打開Discover，選擇 filebeat-* 就能看到日志數據了。

可以看到詳細字段：

參考

1、Logstash使用grok過濾nginx日志（二） - Orgliny - 博客園
https://www.cnblogs.com/Orgliny/p/5592186.html
2、Rsyslog日志服務搭建 - K‘e0llm - 博客園
http://www.cnblogs.com/Eivll0m/p/6700828.html
3、Logstash中如何處理到ElasticSearch的數據映射 - Cocowool - 博客園
https://www.cnblogs.com/cocowool/p/7347069.html
4、ELK 架構之 Logstash 和 Filebeat 安裝配置 - 田園里的蟋蟀 - 博客園
http://www.cnblogs.com/xishuai/p/elk-logstash-filebeat.html
5、搭建ELK日志分析平台（下）—— 搭建kibana和logstash服務器-zero菌-51CTO博客
http://blog.51cto.com/zero01/2082794


免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。



猜您在找 ELK Stack (2) —— ELK + Redis收集Nginx日志 ELK之四-----logstash收集nginx日志及tcp日志 elk收集分析nginx access日志 ELK之八----Logstash結合kafka收集系統日志和nginx日志 ELK日志系統之使用Rsyslog快速方便的收集Nginx日志 ELK：收集k8s容器日志最佳實踐 elk收集windows日志 ELK 6安裝配置 nginx日志收集 kabana漢化 ELK收集監控nginx請求日志 elastalert 報警 ELK日志收集與監控
 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM