通過 logstash 的 tcp/udp 插件收集日志，通常用於在向 elasticsearch 日志補錄丟
失的部分日志，可以將丟失的日志寫到一個文件，然后通過 TCP 日志收集方式直
接發送給 logstash 然后再寫入到 elasticsearch 服務器。
https://www.elastic.co/guide/en/logstash/5.6/input-plugins.html
准備條件：
環境：jdk，安裝好 logstash
安裝nc
[root@es-web2 ~]# apt install nc
安裝jdk
[root@es-web2 ~]# apt install openjdk-8-jdk -y
dpkg安裝
[root@es-web2 src]# dpkg -i logstash-7.12.1-amd64.deb
配置個文件，先進行收集測試
[root@es-web2 ]# vim /etc/logstash/conf.d/tcp-log-es.conf
input{
tcp{
port => 8899
type => "tcplog"
mode => "server"
}
}
output{
stdout{
codec => rubydebug
}
}
驗證
[root@es-web2 ]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tcp-log-es.conf
其他服務器安裝 “瑞士軍刀” nc
[root@es-web2 ]# apt install nc
測試
[root@es-web2 ]# echo "nc test" | nc 172.31.2.107 8899
檢查端口
[root@es-web2 ]# ss -tnl | grep 8899
LISTEN 0 128 *:8899 *:*
測試接收文件
[root@es-web2 ]# nc 172.31.2.107 8899 < /etc/passwd
將輸出改為 elasticsearch
root@long:~# vim /etc/logstash/conf.d/tcp-log-es.conf
input{
tcp{
port => 8899
type => "tcplog"
mode => "server"
}
}
output{
elasticsearch{
hosts => ["172.31.2.101:9200"]
index => "long-tcplog-%{+YYYY.MM.dd}"
}
}
重啟
root@long:~# systemctl restart logstash
再用nc 傳數據
root@long:~# echo "nc test1" | nc 172.31.2.108 8899
root@long:~# echo "偽設備1" > /dev/tcp/172.31.2.108/8899
查看es
添加到kibana
略
logstash收集UDP日志
准備一台CentOS代替交換機
安裝rsyslog和haproxy
[root@localhost ~]# yum install rsyslog
[root@localhost ~]# yum install haproxy -y
rsyslog配置
[ root@localhost ~]# vim /etc/rsyslog.conf
$ModLoad imudp
$UDPServerRun 514
# 最后一行添加
local2.* @@remote-host:514
haproxy配置
[ root@localhost ~]# vim /etc/haproxy/haproxy.cfg
listen web-port
bind 0.0.0.0:80
server 172.31.2.108 172.31.2.108:80 check inter 3s fall 3 rise 5
重啟
[ root@localhost ~]# systemctl restart haproxy
測試網頁，可以訪問即可
配置rsyslog寫入日志
[ root@localhost ~]# vim /etc/rsyslog.conf
#local2.* @@remote-host:514
local2.* /var/log/haproxy.log
重啟
[ root@localhost ~]# systemctl restart rsyslog
改haproxy配置
[ root@localhost ~]# vim /etc/haproxy/haproxy.cfg
listen web-port
bind 0.0.0.0:80
log global
mode http
server 172.31.2.108 172.31.2.108:80 check inter 3s fall 3 rise 5
重啟
[ root@localhost ~]# systemctl restart haproxy
查看日志寫入
[ root@localhost ~]# tail -f /var/log/haproxy.log
配置測試logstash輸出到終端顯示
input{
syslog{
host => "172.31.0.18"
port => "6514"
type => "ststem-rsyslog"
}
}
output {
stdout {}
}
停止
root@long:~# systemctl stop logstash
啟動
root@long:~# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/rsys-log-es.conf
然后在centos-18服務器改rsyslog配置
[ root@localhost ~]# systemctl restart rsyslog
local2.* @@172.31.0.18:6514
重啟
[ root@localhost ~]# systemctl restart rsyslog
啟動
root@long:~# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/rsys-log-es.conf
然后刷新頁面看看機台有沒有獲取到信息，獲取到信息說明是成功的
在haproxy配置添加
[ root@localhost ~]# vim /etc/haproxy/haproxy.cfg
listen web1-port
bind 172.31.2.108:5601
log global
mode tcp
server 172.31.2.101 172.31.2.101:5601 check inter 3s fall 3 rise 5
server 172.31.2.102 172.31.2.102:5601 check inter 3s fall 3 rise 5
重啟
[ root@localhost ~]# systemctl restart haproxy
檢查端口
[ root@localhost ~]# ss -tnl
9200
在上面的基礎上修改配置（Ubuntu的rsyslog日志配置有問題，所有這里使用的CentOS系統）
[root@es-web2 ~]# vim /etc/logstash/conf.d/tcp-log-es.conf
input{
tcp{
port => 8899
type => "tcplog"
mode => "server"
}
syslog {
type => "ststem-rsyslog"
port => "6514"
}
}
#output{
# stdout{
# codec => rubydebug
# }
#}
output{
if [type] == "tcplog" {
elasticsearch {
hosts => ["172.31.2.101:9200"]
index => "long-tcplog-%{+YYYY.MM.dd}"
}}
if [type] == "ststem-rsyslog" {
elasticsearch {
hosts => ["172.31.2.101:9200"]
index => "long-rsyslog-%{+YYYY.MM.dd}"
}}
}
啟動
root@long:~# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/rsys-log-es.conf
訪問幾下網頁，出現下面的即可
然后添加到 kibana
略
如果時間沒有同步，執行下面命令即可
[root@localhost ~]# ntpdate time1.aliyun.com
[root@localhost ~]# hwclock -w
重啟
[ root@localhost ~]# systemctl restart rsyslog
時區不對，執行如下命令即可（CentOS7）
[root@localhost ~]# cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
