1、下載鏡像
這里我們使用elk集成鏡像，地址：https://hub.docker.com/r/sebp/elk/tags
[root@centos-mq ~]# docker pull sebp/elk:660
注：660為elk版本
2、啟動
[root@centos-mq ~]# echo "vm.max_map_count=262144" > /etc/sysctl.conf
[root@centos-mq ~]# sysctl -p
[root@centos-mq ~]# docker run -dit --name elk \ -p 5601:5601 \ -p 9200:9200 \ -p 5044:5044 \ -v /opt/elk-data:/var/lib/elasticsearch \ -v /etc/localtime:/etc/localtime \ sebp/elk:660
說明：-p 指定映射端口，5601kibana訪問，9200es端口，5044 logstash收集日志端口；-v 指定es數據目錄
3、訪問
啟動后等待數據初始化后，瀏覽器輸入：http://10.10.0.13:5601,可看到kibana web界面
4、文件目錄
通過docker exec -it elk /bin/bash可進入容器中，具體各服務配置文件路徑如下
[root@centos-mq ~]# docker exec -it elk /bin/bash /etc/logstash/ ## logstash 配置文件路徑 /etc/elasticsearch/ ##es 配置文件路徑 /var/log/ ## 日志路徑
5、通過filebeat收集java
filebeat部署，版本最好與elk一直，這里也選擇6.6.0版本，filebeat部署在應用所在服務器，進行日志收集，日志樣例；
2019-06-19 14:34:23.261 [http-nio-8090-exec-7] INFO com.one.api.user.controller.UserBacklogController - [李強]:獲取待辦數 2019-06-16 09:36:54.083 [http-nio-8090-exec-6] ERROR com.one.common.exception.OneGlobalExceptionHandler - 【000000系統異常】: URL : http://172.16.223.53:8090/api/backlog/handle ERROR : {} java.lang.NullPointerException: null at com.one.api.contract.service.ContractProcessService.getManagementFieldDataDto(ContractProcessService.java:2394) at com.one.api.contract.service.ContractProcessService.getContractFieldData(ContractProcessService.java:2190) at com.one.api.contract.service.ContractProcessService.leasePass(ContractProcessService.java:1051) at com.one.api.contract.service.ContractProcessService.approvalPass(ContractProcessService.java:530) at com.one.api.contract.service.ContractProcessService$$FastClassBySpringCGLIB$$def3f504.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) at com.one.api.contract.service.ContractProcessService$$EnhancerBySpringCGLIB$$e6bec33f.approvalPass(<generated>) at com.one.api.process.flowable.listener.EndTaskHandler.notify(EndTaskHandler.java:77) at com.one.api.process.flowable.listener.EndTaskHandler$$FastClassBySpringCGLIB$$cc1052a6.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) at com.one.api.process.flowable.listener.EndTaskHandler$$EnhancerBySpringCGLIB$$a8a27165.notify(<generated>) at org.flowable.engine.impl.delegate.invocation.ExecutionListenerInvocation.invoke(ExecutionListenerInvocation.java:35) at org.flowable.engine.impl.delegate.invocation.DelegateInvocation.proceed(DelegateInvocation.java:35) at org.flowable.engine.impl.delegate.invocation.DefaultDelegateInterceptor.handleInvocation(DefaultDelegateInterceptor.java:26) at org.flowable.engine.impl.bpmn.listener.DelegateExpressionExecutionListener.notify(DelegateExpressionExecutionListener.java:45) at org.flowable.engine.impl.bpmn.listener.ListenerNotificationHelper.executeExecutionListeners(ListenerNotificationHelper.java:79) at org.flowable.engine.impl.agenda.AbstractOperation.executeExecutionListeners(AbstractOperation.java:78) at org.flowable.engine.impl.agenda.AbstractOperation.executeExecutionListeners(AbstractOperation.java:69) at org.flowable.engine.impl.agenda.ContinueProcessOperation.executeSynchronous(ContinueProcessOperation.java:141) at org.flowable.engine.impl.agenda.ContinueProcessOperation.continueThroughFlowNode(ContinueProcessOperation.java:113) at org.flowable.engine.impl.agenda.ContinueProcessOperation.continueThroughSequenceFlow(ContinueProcessOperation.java:311) at org.flowable.engine.impl.agenda.ContinueProcessOperation.run(ContinueProcessOperation.java:79) at org.flowable.engine.impl.interceptor.CommandInvoker.executeOperation(CommandInvoker.java:88) at org.flowable.engine.impl.interceptor.CommandInvoker.executeOperations(CommandInvoker.java:72) at org.flowable.engine.impl.interceptor.CommandInvoker.execute(CommandInvoker.java:56) at org.flowable.engine.impl.interceptor.BpmnOverrideContextInterceptor.execute(BpmnOverrideContextInterceptor.java:25) at org.flowable.common.engine.impl.interceptor.TransactionContextInterceptor.execute(TransactionContextInterceptor.java:53) at org.flowable.common.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:71) at org.flowable.common.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:49) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) at org.flowable.common.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:46) at org.flowable.common.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:30)
a）下載安裝
[root@centos-mq ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.6.0-x86_64.rpm
[root@centos-mq ~]# rpm -ivh filebeat-6.6.0-x86_64.rpm
b）配置收集java日志
[root@vanje-dev02 ~]# vim /etc/filebeat/filebeat.yml #=========================== Filebeat inputs ============================ filebeat.inputs:- type: log enabled: true paths: - /apps/oneJars/himalaya/logs/one.log ## 日志路徑 tags: ["one-himalaya"] ## 標簽，用於判斷 multiline.pattern: '^\d{4}-\d{2}-\d{2}' ## 匹配日志開頭 multiline.negate: true ## 日志合並 multiline.match: after
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
#hosts: ["localhost:9200"] ## 注釋 這里是配置采集的日志存放的方式，我們先經過logstash處理，所以這里注釋
#output.elasticsearch:
#hosts: ["localhost:9200"] ## 注釋 這里是配置采集的日志存放的方式，我們先經過logstash處理，所以這里注釋
#----------------------------- Logstash output --------------------------------
output.logstash:
hosts: ["10.10.0.13:5044"] ## 采集日志輸出到logstash，ip為logstash服務ip
output.logstash:
hosts: ["10.10.0.13:5044"] ## 采集日志輸出到logstash，ip為logstash服務ip
## 啟動filebeat
[root@vanje-dev02 ~]# systemctl restart filebeat
c）logstash配置
以下配置只是收集"2019-06-19 14:34:23.261 [http-nio-8090-exec-7] INFO com.one.api.user.controller.UserBacklogController - [李強]:獲取待辦數"格式日志，用來分析用戶使用時間及姓名
## 定過濾指定日志，沒用的日志我們這里不收集，正常是要收集所有，
vim /etc/logstash/conf.d/02-beats-input.conf input { beats { port => 5044 } } filter { #if "one-himalaya" in [tags] { if [message] =~ '獲取待辦數' { grok { patterns_dir => ["/etc/logstash/patterns"] match => { "message" => "%{DAYTIME:day}\s*%{THREAD:thread}\s%{LOGEVL:level}\s*%{JAVACLASS:class}\s*- \[%{JAVALOGMESSAGE:logmessage}\]:" } } } } output { # if "one-himalaya" in [tags] { if [message] =~ '獲取待辦數' { elasticsearch { hosts => ["172.16.223.55:9200"] index => "one-himalaya10-15-%{+YYYY.MM.dd}" } } }
## 自定義匹配規則
vim /etc/logstash/patterns
DAYTIME \d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{1,2}.\d{1,4}
LOGEVL [a-zA-Z]{4}
NUM \d{1}
THREAD \[http-nio-\d{4}-exec-\d{1,3}\]
## 重啟logstash
/etc/init.d/logstash restart
說明：實際生產中，我們會對收集的日志做各種分析提取，需要靈活使用grok 正則匹配來提取自己想要的數據。 可以通過Kibana Dev Tools>>Grok Debugger工具進行調式，例如：
6、kibana查看日志
打開kibana web界面，跟據提示創建索引，即可看到已經收集的日志信息：day logmessage 這兩個字段，為切割日志自定義的字段，這些字段跟據自己所需靈活定義，以區分切割的數據。
