prometheus 监控jvm（下）

1 jmx exporter client

下载jmx_prometheus_javaagent-0.3.1.jar

启动：

nohup java -javaagent:jmx_prometheus_javaagent-0.3.1.jar=8888:config.yaml -XX:+PrintGCDetails -XX:+PrintGC -XX:+PrintGCTimeStamps -Xmx80m -Xms80m -Xmn40m  -Xloggc:gc.txt  -jar MyTest-1.0-SNAPSHOT-jar-with-dependencies.jar &

 

config.yaml:

---
rules:
- pattern: ".*"

 

代码：

    public static void main(String[] args){
        ScheduledExecutorService ses = Executors.newScheduledThreadPool(1);
        ses.scheduleAtFixedRate(new TimerTask(),0, 10, TimeUnit.SECONDS);
    }

    static class TimerTask implements Runnable {

        @Override
        public void run() {
            {
                System.out.println("run");
                byte[] holder = new byte[32 * 1024 * 1024];
                try {
                    Thread.sleep(5000);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }

                holder = null;
            }
            System.gc();
        }
    }

 

本地：

10s一次gc

run
5.477: [GC (System.gc()) [PSYoungGen: 3686K->496K(35840K)] 36454K->33272K(76800K), 0.0019729 secs] [Times: user=0.00 sys=0.01, real=0.00 secs] 
5.479: [Full GC (System.gc()) [PSYoungGen: 496K->0K(35840K)] [ParOldGen: 32776K->431K(40960K)] 33272K->431K(76800K), [Metaspace: 3392K->3392K(1056768K)], 0.0075225 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
run
15.463: [GC (System.gc()) [PSYoungGen: 8641K->1536K(35840K)] 41840K->34743K(76800K), 0.0054047 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
15.468: [Full GC (System.gc()) [PSYoungGen: 1536K->0K(35840K)] [ParOldGen: 33207K->1870K(40960K)] 34743K->1870K(76800K), [Metaspace: 8653K->8653K(1056768K)], 0.0203323 secs] [Times: user=0.05 sys=0.00, real=0.02 secs] 
run
25.464: [GC (System.gc()) [PSYoungGen: 2256K->96K(35840K)] 36894K->34742K(76800K), 0.0013583 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
25.465: [Full GC (System.gc()) [PSYoungGen: 96K->0K(35840K)] [ParOldGen: 34646K->1885K(40960K)] 34742K->1885K(76800K), [Metaspace: 8778K->8778K(1056768K)], 0.0438101 secs] [Times: user=0.05 sys=0.00, real=0.04 secs] 
run
35.464: [GC (System.gc()) [PSYoungGen: 2459K->96K(35840K)] 37112K->34757K(76800K), 0.0014389 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
35.465: [Full GC (System.gc()) [PSYoungGen: 96K->0K(35840K)] [ParOldGen: 34661K->1887K(40960K)] 34757K->1887K(76800K), [Metaspace: 8845K->8845K(1056768K)], 0.0128475 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
run
45.464: [GC (System.gc()) [PSYoungGen: 2254K->96K(35840K)] 36909K->34759K(76800K), 0.0008053 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
45.465: [Full GC (System.gc()) [PSYoungGen: 96K->0K(35840K)] [ParOldGen: 34663K->1888K(40960K)] 34759K->1888K(76800K), [Metaspace: 8880K->8880K(1056768K)], 0.0347415 secs] [Times: user=0.05 sys=0.00, real=0.03 secs] 

visual vm 

 

https://www.jianshu.com/p/adada9c1f7dd

https://github.com/prometheus/jmx_exporter/blob/master/README.md

 

2 server

 43   - job_name: 'jmx'
 44     scrape_interval: 1s
 45     static_configs:
 46     - targets: ['192.168.57.67:8888']
 47       labels:
 48         instance: java_test

 

jvm_memory_bytes_used of prom

 

 

58904.139: [Full GC (Ergonomics) [PSYoungGen: 39936K->0K(40448K)] [ParOldGen: 39452K->6784K(40960K)] 79388K->6784K(81408K), [Metaspace: 9581K->9581K(1058816K)], 0.0056723 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
58905.138: [GC (System.gc()) [PSYoungGen: 2927K->256K(40448K)] 9711K->7040K(81408K), 0.0023703 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
58905.140: [Full GC (System.gc()) [PSYoungGen: 256K->0K(40448K)] [ParOldGen: 6784K->6650K(40960K)] 7040K->6650K(81408K), [Metaspace: 9581K->9581K(1058816K)], 0.0186146 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
58914.138: [Full GC (Ergonomics) [PSYoungGen: 39936K->0K(40448K)] [ParOldGen: 39418K->6770K(40960K)] 79354K->6770K(81408K), [Metaspace: 9581K->9581K(1058816K)], 0.0090674 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
58915.137: [GC (System.gc()) [PSYoungGen: 4420K->288K(40448K)] 11190K->7058K(81408K), 0.0009626 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
58915.138: [Full GC (System.gc()) [PSYoungGen: 288K->0K(40448K)] [ParOldGen: 6770K->6684K(40960K)] 7058K->6684K(81408K), [Metaspace: 9581K->9581K(1058816K)], 0.0078362 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
58924.137: [Full GC (Ergonomics) [PSYoungGen: 39936K->0K(40448K)] [ParOldGen: 39452K->6783K(40960K)] 79388K->6783K(81408K), [Metaspace: 9581K->9581K(1058816K)], 0.0053898 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
58925.137: [GC (System.gc()) [PSYoungGen: 3666K->256K(40448K)] 10450K->7039K(81408K), 0.0004041 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
58925.137: [Full GC (System.gc()) [PSYoungGen: 256K->0K(40448K)] [ParOldGen: 6783K->6684K(40960K)] 7039K->6684K(81408K), [Metaspace: 9581K->9581K(1058816K)], 0.0064194 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]

 

 

1 可以看出本地与线上都是10s一次gc

2 本地和线上内存有一定差异，但内存周期基本一致

 

 

3 dashbard

推荐3066和3457

3457的json需要注意：

label_values(up{job='jmx'}, instance)

 

这里好像只能监控一个job，这里的job对应于server的job_name

另外两个dash好像对于10s级别的内存都没有很好的输出