Centos7.x RPM安裝ELK 7.5.0

一、環境介紹

單位需要分析tomcat 日志和業務日志，比較以后還是選擇用ELK 來進行日志的分析，以及可視化的展示。

系統環境

服務器：

1、AWS EC2 2C8G

[root@ip-10-0-10-229 ~]cat /etc/redhat-release 
CentOS Linux release 7.7.1908 (Core)
[root@ip-10-0-10-229 ~]uname -a
Linux elk-server 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

 

2、JDK版本

jdk版本要求9版本以上的

[root@ip-10-0-10-229 ~]java  -version
openjdk version "13.0.1" 2019-10-15
OpenJDK Runtime Environment AdoptOpenJDK (build 13.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 13.0.1+9, mixed mode, sharing)

 

3、ELK 版本

elasticsearch 7.5.0
kibana 7.5.0
logstash 7.5.0

 

4、JDK 安裝包

#附上一個JAVA 下載地址，當然你也可以不用，elasticsearch7.5中包含有java所以只需要設置變量即可
wget https://download.java.net/java/GA/jdk11/13/GPL/openjdk-11.0.1_linux-x64_bin.tar.gz
/etc/profile 
#文末添加以下配置
export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE HISTCONTROL
export JAVA_HOME=/usr/share/elasticsearch/jdk
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar 
#使變量生效
source /etc/profile 

 

5、ELK 安裝包

官網的包下載實在是太慢，這個包是在aws 的S3桶里的

wget https://rgc-solution-server-validation.s3.cn-north-1.amazonaws.com.cn/xuewenlong/elasticsearch-7.5.0-x86_64.rpm
wget https://rgc-solution-server-validation.s3.cn-north-1.amazonaws.com.cn/xuewenlong/kibana-7.5.0-x86_64.rpm
wget https://rgc-solution-server-validation.s3.cn-north-1.amazonaws.com.cn/xuewenlong/logstash-7.5.0.rpm

 

二、elasticsearch安裝

1、安裝elasticsearch

rpm -ivh elasticsearch-7.5.0-x86_64.rpm

 

2、修改elasticsearch配置文件

[root@ip-10-0-10-229 ~]# cat /etc/elasticsearch/elasticsearch.yml |grep  -v "^#"
#數據存儲路徑
path.data: /var/lib/elasticsearch
#日志存儲路徑
path.logs: /var/log/elasticsearch
#服務端口
http.port: 9200
#集群名
cluster.name: elk-cluster
#node名
node.name: elk-1
#集群master需要和node名設置一致
cluster.initial_master_nodes: ["node-1"]
network.host: 10.0.10.229
 
#xpack密碼配置
xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
 
#elasticsearch-head插件
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-methods: OPTIONS, HEAD, GET, POST, PUT, DELETE
http.cors.allow-headers: "X-Requested-With, Content-Type, Content-Length, X-User"
 
#可選優化配置
#設置單個request請求的內存熔斷限制，默認是jvm堆的60%（es7.0引入了新的內存熔斷機制，會智能判斷，規避OOM）。
indices.breaker.request.limit: 10%
#query請求可使用的jvm內存限制，默認是10%。
indices.queries.cache.size: 20%
#查詢request請求的DSL語句緩存，被緩存的DSL語句下次請求時不會被二次解析，可提升檢索性能，默認值是1%。
indices.requests.cache.size: 2%
#設置字段緩存的最大值，默認無限制。
indices.fielddata.cache.size: 30%
#用來對索引數據進行冷熱分離，需要注意的是 setting 中也要進行相關配置
#"index.routing.allocation.require.box_type": "hot"
node.attr.box_type: hot

 

重點

3、設置elasticsearch的jave目錄

手動安裝java需設置

使用es自帶的java無需設置，版本的問題不知道是不是因為一開始用JDK8 沒有設置路徑的問題，可測試下

4、修改配置文件

[root@ip-10-0-10-229 ~]cat /etc/sysconfig/elasticsearch |grep  JAVA

 

5、X-Pack設置密碼訪問

[root@ip-10-0-10-229 elasticsearch]# cat /etc/elasticsearch/elasticsearch.yml |grep  -v "^#"
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
http.port: 9200
cluster.name: elk-cluster
node.name: elk-1
cluster.initial_master_nodes: ["node-1"]
network.host: 10.0.10.229
 
xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
 
[root@ip-10-0-10-229 elasticsearch]# systemctl  restart elasticsearch
[root@ip-10-0-10-229 elasticsearch]# systemctl  status  elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2020-06-08 01:51:52 UTC; 7s ago
     Docs: http://www.elastic.co
 Main PID: 5453 (java)
   CGroup: /system.slice/elasticsearch.service
           ├─5453 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.enco...
           └─5548 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
 
Jun 08 01:51:37 ip-10-0-10-229.cn-north-1.compute.internal systemd[1]: Starting Elasticsearch...
Jun 08 01:51:37 ip-10-0-10-229.cn-north-1.compute.internal elasticsearch[5453]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely... release.
Jun 08 01:51:52 ip-10-0-10-229.cn-north-1.compute.internal systemd[1]: Started Elasticsearch.
Hint: Some lines were ellipsized, use -l to show in full.
 
[root@ip-10-0-10-229 elasticsearch]# /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y
 
 
Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana]:
Reenter password for [kibana]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

 

5、啟動 elasticsearch

systemctl start elasticsearch
systemctl enable elasticsearch

 

6、檢測是否啟動

elasticsearch 啟動之后有時候會退出，這個時候有檢查下內存是否夠用或者適當的增加配置

[root@ip-10-0-10-229 ~]# netstat -pntl |grep  java 
tcp6    0    0   10.0.10.229:9200    :::*     LISTEN   13898/java          
tcp6    0    0   10.0.10.229:9300    :::*     LISTEN   13898/java 
[root@ip-10-0-10-229 ~]curl 10.0.10.229:9200
{
  "name" : "node-1",
  "cluster_name" : "my-es",
  "cluster_uuid" : "FhHOQO2MQbWRX0MiTRFF6g",
  "version" : {
    "number" : "7.5.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "e9ccaed468e2fac2275a3761849cbee64b39519f",
    "build_date" : "2019-11-26T01:06:52.518245Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

 

7、安裝elasticsearch-head插件

在elasticsearch.yml文末中增加以下配置解決跨域訪問的問題

http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-methods: OPTIONS, HEAD, GET, POST, PUT, DELETE
http.cors.allow-headers: "X-Requested-With, Content-Type, Content-Length, X-User"
 
git clone git://github.com/mobz/elasticsearch-head.git
cd elasticsearch-head
npm install
npm run start
open http://localhost:9100/

 

二、安裝kibana

1、安裝 kibana

rpm -ivh kibana-7.5.0-x86_64.rpm

 

2、修改配置文件

[root@ip-10-0-10-229 ~]# cat /etc/kibana/kibana.yml |grep  -v "^#"
server.port: 5601
server.host: "10.0.10.229"
logging.dest: /var/log/kibana/kibana.log 
elasticsearch.hosts: ["http://10.0.10.229:9200/"]
kibana.index: ".kibana"
elasticsearch.username: "kibana"
elasticsearch.password: "bsh@123"
i18n.locale: "zh-CN"

 

重點

7版本elasticsearch.hosts的配置在6版本里面為elasticsearch.url 需要配置正確否則message日志會報錯

FATAL  Error: [elasticsearch.url]: definition for this key is missing

 

3、啟動

systemctl start kibana
systemctl enable kibana

 

web頁面查看，還沒有索引

三、logstash安裝配置

1、安裝logstash

rpm -vih logstash-7.5.0.rpm

 

2、寫一個配置文件收集系統日志

[root@ip-10-0-10-229 ~]cat /etc/logstash/conf.d/file.conf 
 input{
  file{
    path => ["/var/log/messages"]
    type => "system-log"
    start_position => "beginning"
  }
}
filter{
}
output{
  elasticsearch{
    hosts => ["10.0.10.229:9200"]
    index => "system-log-%{+YYYY.MM}"
    user => "elastic"
    password => "xuewenlong@123"
  }
}

 

3、啟動放置在后台

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/file.conf &

 

4、把日志添加至kiban展示

 

 

ELK7 搭建完成

 

四、收集tomcat access 日志

登錄tomcat server下載logstash安裝包

1、安裝 logstash

wget https://rgc-solution-server-validation.s3.cn-north-1.amazonaws.com.cn/xuewenlong/logstash-7.5.0.rpm
rpm -i  logstash-7.5.0.rpm 

 

2、添加日志文件

[root@ip-tomcat ~]#cat /etc/logstash/conf.d/miniprogram-prod-access-bz.conf
 input{
  file{
    path => ["/home/bsh/tools/apache-tomcat-8.5.23/logs/localhost_access_log*.log"]
    type => "access"
    start_position => "beginning"
    codec => "json"
  }
 
  file{
    path => ["/home/ec2-user/homeconnect/logs/AspectLog/aspect.log"]
    type => "aspect"
    start_position => "beginning"
    codec => "json"
  }
}
 
filter{
mutate {
    convert => ["Request time", "float"]
    }
        if [ip] != "-" {
        geoip {
                       source => "ip"
                        target => "geoip"
                       # database => "/usr/share/GeoIP/GeoIPCity.dat"
                        add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
                        add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
                }
                   mutate {
                        convert => [ "[geoip][coordinates]", "float"]
                }
        }
 }
output{
 
  if [type] == "aspect" {
    elasticsearch {
    hosts => ["10.0.10.229:9200"]
    index => "logstash-miniprogram-uat-aspect-bz.log.%{+YYYY.MM}"
    user => "elastic"
    password => "xuewenlong@123"
        }
    }
  if [type] == "access" {
    elasticsearch {
    hosts => ["10.0.10.229:9200"]
    index => "logstash-miniprogram-uat-access-bz.log.%{+YYYY.MM}"
    user => "elastic"
    password => "xuewenlong@123"
        }
  }
}

 

3、添加索引

 

4、登錄tomcat 頁面測試下日志

 

 

  xuewenlong|雪文龍