java性能分析 - CPU飆高分析工具

背景

        有處理過生產問題的同學基本都能遇到系統忽然緩慢，CPU突然飆升，甚至整個應用請求不可用。當出現這種情況下，在不影響數據准確性的前提下，我們應該盡快導出jstack和內存信息，然后重啟系統，盡快回復系統的可用性，避免用戶體驗過差。本文針對CPU飆升問題，提供該問題的排查思路，從而能夠快速定位到某線程甚至某快代碼導致CPU飆升，從而提供處理該問題的思路。

排查過程

1. 通過top命令查看cpu飆升的java進程pid
2. 通過ps -mp [pid] -o THREAD,tid,time查看該進程下所擁有的線程及各個線程占用cpu的使用率，並且記錄CPU使用率過高的線程ID號
3. 將線程ID號轉換為16進程的數值記為tid_hex
4. 使用jdk自帶jstack監控命令
5. 使用命令jstack [pid] | grep tid_hex -A100命令輸出該線程的堆棧信息
6. 根據堆棧信息分析代碼。

通過以上步驟可以查找出導致cpu飆升的相關代碼位置，然后對代碼進行code review即可。

工具封裝

1. 以上步驟已經封裝為腳本文件,通過以下腳本文件只需要指定進程ID即pid即可導出默認前5條導致CPU率過高的堆棧信息。
2. 已上傳github : 點我進入

./java-thread-top.sh -p pid

#!/bin/bash
# @Function
# Find out the highest cpu consumed threads of java processes, and print the stack of these threads.
# @github https://github.com/cjunn/script_tool/
# @author cjunn
# @date Sun Jan 12 2020 21:08:58 GMT+0800
#

pid='';
count=5;
mode=0;
function usage(){
	readonly PROG="`basename $0`"
	cat <<EOF
Usage: ${PROG} [OPTION]
Find out the highest cpu consumed threads of java processes,
and print the stack of these threads.
Example:
  ${PROG} -p <pid> -c 5      # show top 5 busy java threads info
Output control:
  -p, --pid <java pid>      find out the highest cpu consumed threads from
                            the specified java process.
                            default from all java process.
  -t, --Top					Using top command to get CPU utilization
  -c, --count <num>         set the thread count to show, default is 5.
Miscellaneous:
  -h, --help                display this help and exit.
EOF
}

#1.Collect script parameters
#2.Check whether PID exists
if [ $# -gt 0 ];
then
	while true; do
		case "$1" in
		-c|--count)
			count="$2"
			shift 2
			;;
		-p|--pid)
			pid="$2"
			shift 2
			;;
		-t|--top)
			mode=1
			shift 1
			;;
		-h|--help)
			usage
			exit 0;
			;;
		--)
			shift 1;
			break
			;;
		*)
			shift 1;
			if [ -z "$1" ] ; then
				break
			fi
			;;
		esac
	done
fi
if  [ ! -n "$pid" ] ;then
	echo "error: -p is empty"
	exit 1;
fi

if [ `jps |grep $pid |wc -l` -ne 1 ];then 
	echo "error: -p is wrong"
	exit 1;
fi

function workerByJstack(){
	local tid_hex=$(printf "%x" $tid);
	echo "====================== tid:${tid}  tid_hex:${tid_hex}  cpu:${cpu}  time:${time} ======================";
	jstack $pid | awk 'BEGIN {RS = "\n\n+";ORS = "\n\n"} /'${tid_hex}'/ {print $0}';
	echo "";
}

function workerByTop(){
	top -Hp $pid -n1 | sed '1,7d'| sed '$d' |sed '$d' | sort -k 10 -n -r | sed $[$count+1]',$d' | awk '{print $10,$2,$12}' | while read cpu tid time
	do
		workerByJstack $pid $tid $cpu $time
	done
}

function workerByPs(){
	#1.Query all threads according to PID.
	#2.Delete header and first line information.
	#3.According to the second column of CPU to sort, reverse display.
	#4.Delete the count + 1 to last column based on the count value.
	#5.Get CPU utilization, TID value, thread used time, and assign them to CPU, TID, time respectively.
	#6.Perform hex conversion on TID.
	#7.Use JDK to monitor all threads of jstack output PID.
	#8.Use awk to regularly query the thread information of tid_hex required.
	#9.Display the stack information of count before thread busy.
	ps -mp $pid -o THREAD,tid,time | sed '1,2d' | sort  -k 2 -n -r |sed $[$count+1]',$d' | awk '{print $2,$8,$9}' | while read cpu tid time
	do
		workerByJstack $pid $tid $cpu $time
	done
}

function worker(){
	echo "start-mode:$mode"
	if [ $mode -eq 0 ];then
		workerByPs
	elif [ $mode -eq 1 ];then
		workerByTop
	fi
}
worker