Kubernetes下的應用監控解決方案

所謂應用監控，更多的是基於java jvm的監控，因為公司運行的中間件大部分都是基於tomcat,Springboot,SpringCloud,當然也必須支持WebLogic.在Kubernetes現有方案中，監控那塊主要是通過cAdvisor,Heapster的組件獲取Pod消耗的memory,CPU和網絡的信息，但如果需要更深入的了解Pod中運行的應用的信息就基本沒有提供缺省的方案。

那么到底應用監控涉及什么的指標，我整理一下大致包括：

• JVM Heap
• JVM Non Heap Memory
• GC
• Thread montoring
• DataSource
• Transactions Per Seconds
• Throughput 等等等等。。。

我這篇是基於JMX+InfluxDB+Grafana的嘗試。

• 1.整體架構

 

 

主要技術點在於：

• 在tomcat下打開jmx監控選項，暴露另一個端口35135
• 通過jmxtrans基於JSON文件獲取需要的jmx數據，然后將數據傳入InfluxDB
• 基於Grafana選擇InfluxDB數據進行展現。

 

 

• 2.實施步驟
• 2.1 鏡像准備

InfuxDB和Grafana都基於Pod方式部署，首先拉取鏡像

docker pull docker.io/influxdb:1.4.2
docker pull docker.io/grafana/grafana:4.6.3

 

[root@k8s-master tomcatjmx]# cat grafana.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: grafana
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: "grafana"
    spec:
      containers:
      - name: grafana 
        image: docker.io/grafana/grafana:4.6.3
        ports:
        - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: grafanasvc
  labels:
    app: grafana
spec:
  ports:
  - port: 3000
    protocol: TCP
    targetPort: 3000
    name: http
  type: NodePort
  selector:
    app: grafana

 

[root@k8s-master tomcatjmx]# cat influxdb.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: influxdb
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: "influxdb"
    spec:
      containers:
      - name: influxdb
        image: docker.io/influxdb:1.4.2
        ports:
        - containerPort: 8086
          name: influxdbport
        - containerPort: 8083
          name: influxadmin
---
apiVersion: v1
kind: Service
metadata:
  name: influxdbsvc
  labels:
    app: influxdb
spec:
  ports:
  - port: 8086
    protocol: TCP
    targetPort: 8086
    name: http
  - port: 8083
    protocol: TCP
    targetPort: 8083
    name: admin
  type: NodePort
  selector:
    app: influxdb

 

• 2.2 應用鏡像構建

整個結構如下：

 jmx的打開問題

首先需要修改catalina.sh,打開jmx選項

JAVA_OPTS="$JAVA_OPTS $JSSE_OPTS -Xms512m -Xmx1024m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=35135"

 

jmx的獲取問題

 下載jmxtrans-all.jar包

 jmxtrans運行命令如下：

java -Djmxtrans.log.dir='/usr/local/log' -jar /usr/local/jmxtrans-268-all.jar -j /usr/local/

其中-j指到需要獲取指標的配置文件，比如tomcat.json文件。

[root@node1 caas]# cat tomcat.json
{
  "servers" : [ {
    "port" : "35135",
    "host" : "HOSTNAME",
    "queries" : [
      {
        "obj" : "java.lang:type=Memory",
        "attr" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ],
        "resultAlias":"jvmMemory",
        "outputWriters" : [ {
          "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
          "url" : "http://influxdbsvc:8086/",
          "username" : "root",
          "password" : "root",
          "database" : "caasdb"
        }]
      },
      {
        "obj" : "java.lang:type=Threading",
        "attr" : [ "ThreadCount", "PeakThreadCount", "TotalStartedThreadCount", "DaemonThreadCount"],
        "resultAlias":"jvmThreading",
        "outputWriters" : [ {
          "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
          "url" : "http://influxdbsvc:8086/",
          "username" : "root",
          "password" : "root",
          "database" : "caasdb"
        }]
      },
      {
        "obj" : "java.lang:type=Memory",
        "attr" : [ "HeapMemoryUsage"],
        "resultAlias":"HeapMemoryUsage",
        "outputWriters" : [ {
          "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
          "url" : "http://influxdbsvc:8086/",
          "username" : "root",
          "password" : "root",
          "database" : "caasdb"
        }]
      },
      {
        "obj" : "Catalina:type=GlobalRequestProcessor,name=\"http-apr-8080\"",
        "attr" : [ "bytesSent" ],
        "resultAlias":"tomcatByteSent",
        "outputWriters" : [ {
          "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
          "url" : "http://influxdbsvc:8086/",
          "username" : "root",
          "password" : "root",
          "database" : "caasdb"
        }]
      },
      {
        "obj" : "java.lang:name=CMS Old Gen,type=MemoryPool",
        "resultAlias": "cmsoldgen",
        "attr" : [ "Usage" ],
        "outputWriters" : [ {
          "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
          "url" : "http://influxdbsvc:8086/",
          "username" : "root",
          "password" : "root",
          "database" : "caasdb"
        }]
      },
      {
        "obj" : "java.lang:type=GarbageCollector,name=*",
        "resultAlias": "gc",
        "attr" : [ "CollectionCount", "CollectionTime" ],
        "outputWriters" : [ {
          "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
          "url" : "http://influxdbsvc:8086/",
          "username" : "root",
          "password" : "root",
          "database" : "caasdb"
        }]
      },
      {
        "obj" : "Catalina:type=Engine",
        "attr" : [ "backgroundProcessorDelay"],
        "resultAlias":"EngineDelay",
        "outputWriters" : [ {
          "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
          "url" : "http://influxdbsvc:8086/",
          "username" : "root",
          "password" : "root",
          "database" : "caasdb"
        }]
      }
    ]
  }]
}

詳解

"outputWriters" : [ {
          "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
          "url" : "http://influxdbsvc:8086/",
          "username" : "root",
          "password" : "root",
          "database" : "caasdb"

這一段主要是訪問influxdb的服務，同時創建caasdb數據庫，用戶名密碼都是root

"servers" : [ {
    "port" : "35135",
    "host" : "HOSTNAME",

port主要是tomcat暴露出來的jmx端口

host是指tomcat位於的主機位置或者主機名稱

 

在容器中運行多個進程問題

這樣jmxtrans會啟動另一個進程，也就是說在這個Pod中會同時啟動倆個進程。所以我們要通過supervisor來啟動。

supervisor配置

[root@node1 caas]# cat supervisord.conf 
[supervisord]
nodaemon=true

[program:tomcat]
command=/usr/local/apache-tomcat-8.5.6/bin/catalina.sh run

[program:jmxtrans]
command=/usr/local/updatejson.sh

為什么不是直接java -jar而是一個叫updatejson.sh的腳本呢？

因為我們需要在這個腳本中將配置json文件的host字段替換成為Pod的主機名或者ip

[root@node1 caas]# cat updatejson.sh 
#!/bin/bash

sed -i "s/HOSTNAME/${HOSTNAME}/" /usr/local/tomcat.json

java -Djmxtrans.log.dir='/usr/local/log' -jar /usr/local/jmxtrans-268-all.jar -j /usr/local/

 

Supervisord不顯示tomcat日志問題

需要在supervisord.conf下加入loglevel和redirect_stderr選項

[root@node1 caas]# cat supervisord.conf 
[supervisord]
nodaemon=true
loglevel=debug

[program:tomcat]
command=/usr/local/apache-tomcat-8.5.6/bin/catalina.sh run
stdout_logfile=/usr/local/apache-tomcat-8.5.6/logs/catalina.log
redirect_stderr=true

[program:jmxtrans]
command=/usr/local/updatejson.sh

 

基本OK了，整個Dockerfile如下

FROM linux7-jre:8u151

RUN mkdir -p "/usr/local"

COPY caas/*.*  /usr/local/

WORKDIR /usr/local

RUN yum install -y python-setuptools && \
    easy_install supervisor && \
    yum clean all && \
    tar -xvf apache-tomcat-8.5.6.tar.gz && \
    cp /usr/local/catalina.sh /usr/local/apache-tomcat-8.5.6/bin/catalina.sh && \
    mv /usr/local/supervisord.conf /etc/  && \
    chmod +x /usr/local/updatejson.sh 

ENV CATALINA_HOME /usr/local/apache-tomcat-8.5.6
ENV PATH $CATALINA_HOME/bin:$PATH
ENV JAVA_HOME /usr/java/default

WORKDIR $CATALINA_HOME

EXPOSE 8080 35135

CMD ["supervisord"]

 啟動以后看到pod運行

 

 服務如下：

 

 比較核心的需要kubectl logs influxdb-4115439627-rm9zb查看一下，確認有數據寫入。

 

 

• 2.3 Grafana配置

datasource配置

 

新建一個Dashboard

點擊Panel Title,然后選擇編輯Edit

選擇相應的指標，形成報告。

 

 

grafana配置詳情參考下圖

 

• 3.其他問題
• 為什么jmxtrans不放在host中進行對多個tomcat的收集

每個jmxtrans需要指定一個配置文件，在這個配置文件中指定監控的主機名，如果都共用一個的話，那所有的指標都會匯集到一起，如果要區分的話，就必須放到Pod中。

• grafana每次當pod發生變化時，圖標無法自動更新，必須手工配置，這個比較難搞。

• Q. How do I use the second y axis, secondYAxis function does not work

  A. You can switch any series to the second y axis by clicking on the colored line to left of the series name in the legend below the graph. Alternately, use the "Display Styles" > "Series Specific overrides" to define an alias or regex + "Y-axis: 2" to move metrics to the right Axis

 

• 4.例子程序

消耗CPU的小程序

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<%@page import="java.util.*"%>

<%@ page contentType="text/html;charset=windows-1252"%>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=windows-1252"/>
    <title>Random</title>
  </head>
  <html>
    <body>
<h3>
      
      <%
               java.util.Random generator = new java.util.Random();
                generator.ints(1000000, 0, 100).sorted();
      %>
      random sort finished .........
      </h3>
 
    </body>
</html>

 

消耗內存的小程序

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<%@page import="java.util.*"%>

<%@ page contentType="text/html;charset=windows-1252"%>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=windows-1252"/>
    <title>Outofmemory</title>
  </head>
  <html>
    <body>
<h3>
      
      <%
      
            List<String> heapList = new ArrayList<String>();
       
         for (int i=0;i<10000;i++) {
                heapList.add(new String("Shobhna"));
        }
         
      %>
      put 10000 heapList success.........
      </h3>
 
    </body>
</html>