5：ELK+Kafka集群安装

一、架构

1、拓扑

 

二、准备工作

 1、软件信息

 注意：Logstash kafka插件对kafka有版本要求，最高只能使用kafka_2.10-0.10.1.0版本

 

 2、环境规划

 

 

三、安装配置

1、ES集群安装（两台ES安装）

=========================================ELK1-1.225配置
rpm -ivh elasticsearch-6.2.3.rpm
cat /etc/elasticsearch/elasticsearch.yml  |  grep -Ev "^$|^#"
cluster.name: es
node.name: es01
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.system_call_filter: false
network.host: 192.168.1.225
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.1.225", "192.168.1.226"]
discovery.zen.minimum_master_nodes: 1
node.master: true
node.data: true
transport.tcp.compress: true

vim /etc/sysconfig/elasticsearch
JAVA_HOME=/usr/local/jdk1.8.0_131/

chkconfig --add elasticsearch
service elasticsearch start

=========================================ELK2-1.226配置
cat /etc/elasticsearch/elasticsearch.yml  |  grep -Ev "^$|^#"
cluster.name: es
node.name: es02
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.system_call_filter: false
network.host: 192.168.1.226
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.1.225", "192.168.1.226"]
discovery.zen.minimum_master_nodes: 1
node.master: false
node.data: true
transport.tcp.compress: true

vim /etc/sysconfig/elasticsearch
JAVA_HOME=/usr/local/jdk1.8.0_131/

chkconfig --add elasticsearch
service elasticsearch start

=========================================查看ES集群状态
curl -XGET 'http://192.168.1.225:9200/_cluster/health?pretty'  
{
  "cluster_name" : "es",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

 

2、Kibana安装汉化

rpm -ivh kibana-6.2.3-x86_64.rpm
cat /etc/kibana/kibana.yml  | grep -Ev "^$|^#"
server.port: 5601
server.host: "192.168.1.225"
elasticsearch.url: "http://192.168.1.225:9200"    #输入任意一个ES IP即可
logging.dest: /var/log/kibana.log

touch /var/log/kibana.log ; chmod 777 /var/log/kibana.log
git clone https://github.com/anbai-inc/Kibana_Hanization.git
cd Kibana_Hanization/
python main.py /usr/share/kibana/
chkconfig --add kibana
/etc/init.d/kibana start 

 

3、Logstash安装（两台ES安装，生产业务建议ES和Logstash分离）

ln -s /usr/local/jdk1.8.0_131/bin/java  /usr/bin/java
rpm -ivh logstash-6.2.3.rpm
cat /etc/logstash/logstash.yml  | grep -Ev "^$|^#"
path.data: /var/lib/logstash
http.host: "192.168.1.225"    #两台IP不同
path.logs: /var/log/logstash

chkconfig --add logstash
/etc/init.d/logstash start 

 

4、ZK集群安装（三台ZK安装）

===============================================zookeeper安装
cd /usr/local/src
tar xf zookeeper-3.4.10.tar.gz 
mv zookeeper-3.4.10 /usr/local/
ln -s /usr/local/zookeeper-3.4.10/ /usr/local/zookeeper 
cp /usr/local/zookeeper/conf/zoo_sample.cfg  /usr/local/zookeeper/conf/zoo.cfg 
mkdir -p /data/{zookeeper,logs/zookeeper,logs/kafka}
cat /usr/local/zookeeper/conf/zoo.cfg |  grep -Ev "^$|^#"
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper
dataLogDir=/data/logs/zookeeper
clientPort=2181
server.1=192.168.1.227:2888:3888
server.2=192.168.1.228:2888:3888
server.3=192.168.1.229:2888:3888

touch /data/zookeeper/myid | echo  1 > /data/zookeeper/myid   #另外两台配置文件相同，myid改为配置文件中一致
/usr/local/zookeeper/bin/zkServer.sh start   #启动zookeeper
/usr/local/zookeeper/bin/zkServer.sh status   #查看三台zookeeper集群状态，正确的是一主两从
Mode: leader
/usr/local/zookeeper/bin/zkServer.sh status  
Mode: follower
/usr/local/zookeeper/bin/zkServer.sh status  
Mode: follower

===============================================kafka安装
cd /usr/local/src
tar xf kafka_2.10-0.10.1.0.tgz.gz
mv kafka_2.10-0.10.1.0 /usr/local/
ln -s /usr/local/kafka_2.10-0.10.1.0/ /usr/local/kafka 
cat /usr/local/kafka/config/server.properties |  grep -Ev "^$|^#"
broker.id=1    #每台ID不同
listeners=PLAINTEXT://192.168.1.227:9092  
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/logs/kafka
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.1.227:2181,192.168.1.228:2181,192.168.1.229:2181
zookeeper.connection.timeout.ms=6000

#启动服务
/usr/local/kafka/bin/kafka-server-start.sh  -daemon /usr/local/kafka/config/server.properties  #三台服务器都启动

#检查启动情况，默认开启的端口为2181（zookeeper）和9202（kafka）

#测试消息生产和消费
/usr/local/kafka/bin/kafka-topics.sh --create --zookeeper 192.168.1.227:2181 --replication-factor 1 --partition 1 --topic test     #创建topic
/usr/local/kafka/bin/kafka-topics.sh --list --zookeeper 192.168.1.227:2181     #查看创建的topic
/usr/local/kafka/bin/kafka-console-producer.sh --broker-list 192.168.1.227:9092 --topic test   #模拟客户端发送消息
/usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.227:9092 --topic test  --from-beginning    #模拟客户端接收信息
/usr/local/kafka/bin/kafka-topics.sh --delete --zookeeper 192.168.1.227:2181 --topic test    #删除topic

 

5、filebeat安装（所有业务主机）

rpm -ivh filebeat-6.2.3-x86_64.rpm
chkconfig --add filebeat
/etc/init.d/filebeat start 

 

6、filebeat配置收集业务日志

vim /etc/filebeat/filebeat.yml
filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /opt/efang_hr/efanghr_warn.log
  fields:
    log_topics: efang
  max_bytes: 1048576

- type: log
  enabled: true
  paths:
    - /var/log/messages
  fields:
    log_topics: syslog
  max_bytes: 1048576

- type: log
  enabled: true
  paths:
    - /usr/local/tomcat/logs/catalina.out
  fields:
    log_topics: tomcat_efang
  max_bytes: 1048576

output.kafka:
  hosts: ["192.168.1.227:9092","192.168.1.228:9092","192.168.1.229:9092"]
  topic: '%{[fields.log_topic]}'
  compression: gzip
  max_message_bytes: 1000000

chkconfig --add filebeat
/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml   #检查配置文件语法合规性
/etc/init.d/filebeat restart  #重启服务

 

7、配置Logstash从ZK将日志传入ES

cat /etc/logstash/conf.d/logstash.conf
input {
  kafka {
    bootstrap_servers => "192.168.1.227:9092,192.168.1.228:9092，192.168.1.229:9092"
    topics => ["efang","tomcat_efang","syslog"]
    group_id => "test-consumer-group"
    codec => "plain"
    consumer_threads => 1
    decorate_events => true

  }
}

output {
 if[fields][log_topics]=="efang"{
  elasticsearch {
    action => "index"
    index => "efang"
    hosts => ["192.168.1.225:9200","192.168.1.226:9200"]
  }
 }

 if[fields][log_topics]=="tomcat_efang"{
  elasticsearch {
    action => "index"
    index => "tomcat_efang"
    hosts => ["192.168.1.225:9200","192.168.1.226:9200"]
  }
 }

 if[fields][log_topics]=="syslog"{
  elasticsearch {
    action => "index"
    index => "syslog"
    hosts => ["192.168.1.225:9200","192.168.1.226:9200"]
  } 
 }
}
/usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/logstash.conf --config.test_and_exit   #检查配置文件合规性
/etc/init.d/logstash restart    #重启服务

 

8、ZK查看topic和topic的消息队列

# 通过 Zookeeper 去查看
/usr/local/kafka/bin/kafka-console-consumer.sh --zookeeper 192.168.1.227:2181,192.168.1.228:2181,192.168.1.229:2181  --topic efang --from-beginning

# 通过 Kafka 本身去查看
/usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.227:2181,192.168.1.228:2181,192.168.1.229:2181 --topic efang --from-beginning

# 如果不想从头看，就把 --from-beginning 去掉，也就是从最新的开始看

 

9、坑

（1）、ES无法启动

[2018-04-09T10:25:18,400][ERROR][o.e.b.Bootstrap          ] [es01] node validation exception
[2] bootstrap checks failed
[1]: max number of threads [1024] for user [elasticsearch] is too low, increase to at least [4096]
[2]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk

 解决：

vim /etc/elasticsearch/elasticsearch.yml  #默认bootstrap.system_call_filter为true进行检测，所以导致检测失败，失败后直接导致ES不能启动。
bootstrap.system_call_filter: false
vim /etc/security/limits.d/90-nproc.conf   #es可创建的线程太小了，改大点
*          soft    nproc     4096

 （2）、Logstash启动警告和没有启动脚本

which: no java in (/sbin:/bin:/usr/sbin:/usr/bin:/usr/X11R6/bin)
could not find java; set JAVA_HOME or ensure java is in PATH
解决：
ln -s /usr/local/jdk1.8.0_131/bin/java  /usr/bin/java

chmod: cannot access `/etc/default/logstash': No such file or directory
warning: %post(logstash-1:6.2.3-1.noarch) scriptlet failed, exit status 1
解决：
/usr/share/logstash/bin/system-install /etc/logstash/startup.options sysv