【ELK】抓取AWS-ELB日志的logstash配置文件

前言

ELK搭建沒有難度，難的是logstash的配置文件，logstash主要分為三個部分，input，filter和output。

input，輸入源可選的輸入源由很多，詳情見ELK官網，這里我們說s3作為輸入源。

filter，過濾器，logstash可以在input和output中間添加過濾器，可以將數據進行分類、過濾、打標簽等操作，將數據格式化。logstash的核心就在此。

output，輸出。一般是輸出到elasticsearch。

說明：

AWS的ELB日志存儲在S3，可以通過logstash的S3插件獲取，經過過濾器后，輸出到elasticsearch。

ELK的搭建和配置在這里就不說了，看官方文檔就行，這里提供一個logstash的配置文件 ，用於抓取和格式化ELB日志。

input {
	s3 {
        access_key_id => "access_key"
        secret_access_key => "secret_key"
        bucket => "elb_bucket"
        region => "aws_region"
        type => "s3"
    }
}

filter { 
	mutate{
	split => { "message" => " " }
	add_field => {
		"log_time" => "%{[message][0]}"
		}
	add_field => {
		"elb_name" => "%{[message][1]}"
		}
	add_field => {
		"client_ip" => "%{[message][2]}"
		}
	add_field => {
		"t1" => "%{[message][4]}"
		}
	add_field => {
		"t2" => "%{[message][5]}"
		}
	add_field => {
		"t3" => "%{[message][6]}"
		}
	add_field => {
		"elb_code" => "%{[message][7]}"
		}
	add_field => {
		"server_code" => "%{[message][8]}"
		}
	add_field => {
		"getpost" => "%{[message][11]}"
		}
	add_field => {
		"url" => "%{[message][12]}"
		}
	remove_field => [ "message" ]
	}
	mutate {
	convert => { "t1" => "float" }
	convert => { "t2" => "float" }
	convert => { "t3" => "float" }
	convert => { "elb_code" => "integer" }
	convert => { "server_code" => "integer" }
	}
	grok {
		break_on_match => false
		match => { "client_ip" => "%{IPV4:device_ip}" }
		match => { "url" => "%{URIPROTO:url_head}://%{URIHOST:url_destination}:%{POSINT:url_port}%{URIPATH:url_path}(?:%{URIPARAM:url_param})?" }
		match => { "getpost" => "%{WORD:get_post}" }
		remove_field => [ "getpost" ]
	}
	mutate{
	split => { "url_path" => "." }
	add_field => {
		"url_api" => "%{[url_path][0]}"
		}
	add_field => {
		"html_ashx" => "%{[url_path][1]}"
		}
	}
	date {
		match => ["log_time", "ISO8601"]
		target => "log_date"
		add_tag => [ "log_date" ]
		remove_field => [ "log_time" ]
	}
	geoip {
		source => "device_ip"
		add_tag => [ "geoip" ]
		remove_field => [ "client_ip" ]
	}
}
output {
	elasticsearch { 
	hosts => ["xxx.xxx.xxx.xxx:9200"] 
	index => "logstash-s3-%{+YYYY-MM-dd}"
	}
}