观测工具下载地址：https://github.com/iovisor/bcc
bcc观测工具的安装方式：https://github.com/iovisor/bcc/blob/master/INSTALL.md
工具简介：
BCC makes BPF programs easier to write, with kernel instrumentation in C (and includes a C wrapper around LLVM), and front-ends in Python and lua. It is suited for many tasks, including performance analysis and network traffic control.
BCC 是用于创建高效内核跟踪和操作程序的工具包，包括一些有用的工具和示例。它利用了扩展的BPF（Berkeley Packet Filters），正式名称为 eBPF。
使用要求：
1）Linux 内核版本最低 4.1
2）kernel-devel 版本必须和内核版本一致
[root@yang-01 ~]# rpm -qa|grep -i kernel | grep -i devel
kernel-devel-3.10.0-1127.18.2.el7.x86_64
[root@yang-01 ~]# uname -r
3.10.0-1127.el7.x86_64
3）kernel 配置文件相关参数必须按照下面的要求
[root@yang-01 ~]# grep -i BPF /boot/config-`uname -r`
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_NET_CLS_BPF=m
CONFIG_BPF_JIT=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=y
4）对于 CentOS 建议升级到 3.6 及以上版本
5）使用默认的 Python 2.7 即可，不需要升级 Python 3
yum安装
[root
@yang
-
01
~]# cat /etc/redhat-release
CentOS Linux release
7.8
.
2003
(Core)
[root
@yang
-
01
~]# uname -r
3.10
.
0
-
1127
.el7.x86_64
[root
@yang
-
01
~]# yum update kernel
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.cn99.com
* epel: mirrors.yun-idc.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package kernel.x86_64
0
:
3.10
.
0
-
1127.18
.
2
.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
===========================================================================================================================================================================
Package Arch Version Repository Size
===========================================================================================================================================================================
Installing:
kernel x86_64
3.10
.
0
-
1127.18
.
2
.el7 updates
50
M
Transaction Summary
===========================================================================================================================================================================
Install
1
Package
Total download size:
50
M
Installed size:
64
M
Is
this
ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
kernel-
3.10
.
0
-
1127.18
.
2
.el7.x86_64.rpm |
50
MB
00
:
00
:
08
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : kernel-
3.10
.
0
-
1127.18
.
2
.el7.x86_64
1
/
1
Verifying : kernel-
3.10
.
0
-
1127.18
.
2
.el7.x86_64
1
/
1
Installed:
kernel.x86_64
0
:
3.10
.
0
-
1127.18
.
2
.el7
Complete!
[root
@yang
-
01
~]# yum install bcc -y
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.cn99.com
* epel: mirrors.yun-idc.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package bcc.x86_64
0
:
0.10
.
0
-
1
.el7 will be installed
--> Processing Dependency: bcc-tools =
0.10
.
0
-
1
.el7
for
package
: bcc-
0.10
.
0
-
1
.el7.x86_64
--> Processing Dependency: llvm-
private
>=
6.0
.
1
-
0.3
for
package
: bcc-
0.10
.
0
-
1
.el7.x86_64
--> Processing Dependency: libLLVM-
7
-rhel.so(LLVM_7)(64bit)
for
package
: bcc-
0.10
.
0
-
1
.el7.x86_64
--> Processing Dependency: libclangSerialization.so.
7
()(64bit)
for
package
: bcc-
0.10
.
0
-
1
.el7.x86_64
--> Processing Dependency: libclangSema.so.
7
()(64bit)
for
package
: bcc-
0.10
.
0
-
1
.el7.x86_64
--> Processing Dependency: libclangRewrite.so.
7
()(64bit)
for
package
: bcc-
0.10
.
0
-
1
.el7.x86_64
--> Processing Dependency: libclangParse.so.
7
()(64bit)
for
package
: bcc-
0.10
.
0
-
1
.el7.x86_64
--> Processing Dependency: libclangLex.so.
7
()(64bit)
for
package
: bcc-
0.10
.
0
-
1
.el7.x86_64
--> Processing Dependency: libclangFrontend.so.
7
()(64bit)
for
package
: bcc-
0.10
.
0
-
1
.el7.x86_64
--> Processing Dependency: libclangEdit.so.
7
()(64bit)
for
package
: bcc-
0.10
.
0
-
1
.el7.x86_64
--> Processing Dependency: libclangDriver.so.
7
()(64bit)
for
package
: bcc-
0.10
.
0
-
1
.el7.x86_64
--> Processing Dependency: libclangCodeGen.so.
7
()(64bit)
for
package
: bcc-
0.10
.
0
-
1
.el7.x86_64
--> Processing Dependency: libclangBasic.so.
7
()(64bit)
for
package
: bcc-
0.10
.
0
-
1
.el7.x86_64
--> Processing Dependency: libclangAnalysis.so.
7
()(64bit)
for
package
: bcc-
0.10
.
0
-
1
.el7.x86_64
--> Processing Dependency: libclangAST.so.
7
()(64bit)
for
package
: bcc-
0.10
.
0
-
1
.el7.x86_64
--> Processing Dependency: libLLVM-
7
-rhel.so()(64bit)
for
package
: bcc-
0.10
.
0
-
1
.el7.x86_64
--> Running transaction check
---> Package bcc-tools.x86_64
0
:
0.10
.
0
-
1
.el7 will be installed
--> Processing Dependency: python-bcc =
0.10
.
0
-
1
.el7
for
package
: bcc-tools-
0.10
.
0
-
1
.el7.x86_64
--> Processing Dependency: python-netaddr
for
package
: bcc-tools-
0.10
.
0
-
1
.el7.x86_64
--> Processing Dependency: kernel-devel
for
package
: bcc-tools-
0.10
.
0
-
1
.el7.x86_64
---> Package llvm-
private
.x86_64
0
:
7.0
.
1
-
1
.el7 will be installed
--> Running transaction check
---> Package kernel-devel.x86_64
0
:
3.10
.
0
-
1127.18
.
2
.el7 will be installed
---> Package python-bcc.x86_64
0
:
0.10
.
0
-
1
.el7 will be installed
---> Package python-netaddr.noarch
0
:
0.7
.
5
-
9
.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
===========================================================================================================================================================================
Package Arch Version Repository Size
===========================================================================================================================================================================
Installing:
bcc x86_64
0.10
.
0
-
1
.el7 base
567
k
Installing
for
dependencies:
bcc-tools x86_64
0.10
.
0
-
1
.el7 base
394
k
kernel-devel x86_64
3.10
.
0
-
1127.18
.
2
.el7 updates
18
M
llvm-
private
x86_64
7.0
.
1
-
1
.el7 base
23
M
python-bcc x86_64
0.10
.
0
-
1
.el7 base
74
k
python-netaddr noarch
0.7
.
5
-
9
.el7 base
983
k
Transaction Summary
===========================================================================================================================================================================
Install
1
Package (+
5
Dependent packages)
Total download size:
42
M
Installed size:
126
M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(
1
/
6
): bcc-tools-
0.10
.
0
-
1
.el7.x86_64.rpm |
394
kB
00
:
00
:
05
(
2
/
6
): bcc-
0.10
.
0
-
1
.el7.x86_64.rpm |
567
kB
00
:
00
:
05
(
3
/
6
): python-bcc-
0.10
.
0
-
1
.el7.x86_64.rpm |
74
kB
00
:
00
:
00
(
4
/
6
): python-netaddr-
0.7
.
5
-
9
.el7.noarch.rpm |
983
kB
00
:
00
:
00
(
5
/
6
): kernel-devel-
3.10
.
0
-
1127.18
.
2
.el7.x86_64.rpm |
18
MB
00
:
00
:
06
(
6
/
6
): llvm-
private
-
7.0
.
1
-
1
.el7.x86_64.rpm |
23
MB
00
:
00
:
30
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total
1.4
MB/s |
42
MB
00
:
00
:
30
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : llvm-
private
-
7.0
.
1
-
1
.el7.x86_64
1
/
6
Installing : kernel-devel-
3.10
.
0
-
1127.18
.
2
.el7.x86_64
2
/
6
Installing : python-netaddr-
0.7
.
5
-
9
.el7.noarch
3
/
6
Installing : python-bcc-
0.10
.
0
-
1
.el7.x86_64
4
/
6
Installing : bcc-tools-
0.10
.
0
-
1
.el7.x86_64
5
/
6
Installing : bcc-
0.10
.
0
-
1
.el7.x86_64
6
/
6
Verifying : python-netaddr-
0.7
.
5
-
9
.el7.noarch
1
/
6
Verifying : python-bcc-
0.10
.
0
-
1
.el7.x86_64
2
/
6
Verifying : bcc-
0.10
.
0
-
1
.el7.x86_64
3
/
6
Verifying : kernel-devel-
3.10
.
0
-
1127.18
.
2
.el7.x86_64
4
/
6
Verifying : bcc-tools-
0.10
.
0
-
1
.el7.x86_64
5
/
6
Verifying : llvm-
private
-
7.0
.
1
-
1
.el7.x86_64
6
/
6
Installed:
bcc.x86_64
0
:
0.10
.
0
-
1
.el7
Dependency Installed:
bcc-tools.x86_64
0
:
0.10
.
0
-
1
.el7 kernel-devel.x86_64
0
:
3.10
.
0
-
1127.18
.
2
.el7 llvm-
private
.x86_64
0
:
7.0
.
1
-
1
.el7 python-bcc.x86_64
0
:
0.10
.
0
-
1
.el7
python-netaddr.noarch
0
:
0.7
.
5
-
9
.el7
Complete!
[root
@yang
-
01
tools]# pwd
/usr/share/bcc/tools
[root
@yang
-
01
tools]# ls
argdist cachestat dcstat fileslower javagc mountsnoop opensnoop pythoncalls rubystat sslsniff tcpaccept tplist
bashreadline cachetop deadlock filetop javaobjnew mysqld_qslower perlcalls pythonflow runqlat stackcount tcpconnect trace
biolatency capable deadlock.c funccount javastat nfsdist perlflow pythongc runqlen statsnoop tcpconnlat ttysnoop
biosnoop cobjnew doc funclatency javathreads nfsslower perlstat pythonstat runqslower syncsnoop tcpdrop vfscount
biotop cpudist drsnoop funcslower killsnoop nodegc phpcalls reset-trace shmsnoop syscount tcplife vfsstat
bitesize cpuunclaimed execsnoop gethostlatency lib nodestat phpflow rubycalls slabratetop tclcalls tcpretrans wakeuptime
bpflist dbslower ext4dist hardirqs llcstat offcputime phpstat rubyflow sofdsnoop tclflow tcpsubnet xfsdist
btrfsdist dbstat ext4slower javacalls mdflush offwaketime pidpersec rubygc softirqs tclobjnew tcptop xfsslower
btrfsslower dcsnoop filelife javaflow memleak oomkill profile rubyobjnew solisten tclstat tcptracer
|
报错举例1
# 报错举例
[root
@yang
-
01
~]# cachestat
1
3
modprobe: FATAL: Module kheaders not found.
chdir(/lib/modules/
3.10
.
0
-
1127
.el7.x86_64/build): No such file or directory
Traceback (most recent call last):
File
"/usr/share/bcc/tools/cachestat"
, line
96
, in <module>
b = BPF(text=bpf_text)
File
"/usr/lib/python2.7/site-packages/bcc/__init__.py"
, line
325
, in __init__
raise Exception(
"Failed to compile BPF text"
)
Exception: Failed to compile BPF text
# 发现报错，这时需要进行内核版本的查看，确保 kernel-devel 与系统上当前安装的内核版本匹配
[root
@yang
-
01
tools]# rpm -qa|grep -i kernel | grep -i devel
kernel-devel-
3.10
.
0
-
1127.18
.
2
.el7.x86_64
[root
@yang
-
01
tools]# uname -r
3.10
.
0
-
1127
.el7.x86_64
[root
@yang
-
01
tools]# yum install
"kernel-devel-uname-r == $(uname -r)"
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.cn99.com
* epel: mirrors.yun-idc.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package kernel-devel.x86_64
0
:
3.10
.
0
-
1127
.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
===========================================================================================================================================================================
Package Arch Version Repository Size
===========================================================================================================================================================================
Installing:
kernel-devel x86_64
3.10
.
0
-
1127
.el7 base
18
M
Transaction Summary
===========================================================================================================================================================================
Install
1
Package
Total download size:
18
M
Installed size:
38
M
Is
this
ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
kernel-devel-
3.10
.
0
-
1127
.el7.x86_64.rpm |
18
MB
00
:
00
:
06
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : kernel-devel-
3.10
.
0
-
1127
.el7.x86_64
1
/
1
Verifying : kernel-devel-
3.10
.
0
-
1127
.el7.x86_64
1
/
1
Installed:
kernel-devel.x86_64
0
:
3.10
.
0
-
1127
.el7
Complete!
|
报错举例2
报错: 在安装kernel-devel显示 No
package
No
package
kernel-devel-uname-r ==
5.6
.
2
-
1
.el7.elrepo.x86_64 available.
解决办法: 启用[elrepo-kerne]存储库,
# vim /etc/yum.repos.d/elrepo.repo
[elrepo-kernel]
name=ELRepo.org Community Enterprise Linux Kernel Repository - el7
baseurl=http:
//elrepo.org/linux/kernel/el7/$basearch/
http:
//mirrors.coreix.net/elrepo/kernel/el7/$basearch/
http:
//mirror.rackspace.com/elrepo/kernel/el7/$basearch/
http:
//repos.lax-noc.com/elrepo/kernel/el7/$basearch/
http:
//mirror.ventraip.net.au/elrepo/kernel/el7/$basearch/
mirrorlist=http:
//mirrors.elrepo.org/mirrors-elrepo-kernel.el7
# 修改这一行将
0
改为
1
enabled=
1
然后重新执行:
# sudo yum install
"kernel-devel-uname-r == $(uname -r)"
|
缓存命中查询 命令使用举例
[root
@yang
-
01
tools]# cachestat
1
5
HITS MISSES DIRTIES HITRATIO BUFFERS_MB CACHED_MB
1409
0
77
100.00
%
1
2440
1438
0
101
100.00
%
1
2440
1413
0
68
100.00
%
1
2440
1412
0
117
100.00
%
1
2440
1411
0
72
100.00
%
1
2440
第一个参数为间隔秒，第二个参数为输出次数
TOTAL ，表示总的 I/O 次数
MISSES ，表示缓存未命中的次数
HITS ，表示缓存命中的次数
DIRTIES， 表示新增到缓存中的脏页数
BUFFERS_MB 表示 Buffers 的大小，以 MB 为单位
CACHED_MB 表示 Cache 的大小，以 MB 为单位
[root
@yang
-
01
tools]# cachetop
16
:
24
:
15
Buffers MB:
1
/ Cached MB:
2443
/ Sort: HITS / Order: ascending
PID UID CMD HITS MISSES DIRTIES READ_HIT% WRITE_HIT%
1653
actionte ustats
1
0
0
100.0
%
0.0
%
1778
actionte umon-mgr
1
0
0
100.0
%
0.0
%
903
mysql jsvc
1
0
0
100.0
%
0.0
%
10909
actionte mysqld
2
0
0
100.0
%
0.0
%
22506
actionte uproxy
2
0
1
50.0
%
0.0
%
869
mysql jsvc
2
2
2
0.0
%
0.0
%
10788
actionte mysqld
3
0
0
100.0
%
0.0
%
1724
actionte urman-agent
4
2
2
33.3
%
0.0
%
21780
actionte uguard-agent
4
2
2
33.3
%
0.0
%
12798
actionte ustats
6
0
0
100.0
%
0.0
%
12799
actionte ustats
6
0
0
100.0
%
0.0
%
12800
actionte ustats
6
0
0
100.0
%
0.0
%
这个命令的输出和top类似，默认按照缓存的命中次数（HITS）排序，展示了每个进程的缓存命中情况。
这里的 HITS、MISSES 和 DIRTIES ，跟 cachestat 里的含义一样，分别代表间隔时间内的缓存命中次数、未命中次数以及新增到缓存中的脏页数。
而 READ_HIT 和 WRITE_HIT ，分别表示读和写的缓存命中率。
|
BCC工具
# 查看新的进程。那些会消耗系统资源，但很短暂的进程，它们甚至不会出现在 top(
1
) 命令或其它工具中的显示之中。这些新进程可以使用 execsnoop 进行检测（或使用行业术语说，可以被追踪traced）。
[root
@yang
-
01
tools]# execsnoop -h
usage: execsnoop [-h] [-t] [-x] [-q] [-n NAME] [-l LINE] [--max-args MAX_ARGS]
Trace exec() syscalls
optional arguments:
-h, --help show
this
help message and exit
-t, --timestamp include timestamp on output
-x, --fails include failed exec()s
-q, --quote Add quotemarks (") around arguments.
-n NAME, --name NAME only print commands matching
this
name (regex), any
arg
-l LINE, --line LINE only print commands where arg contains
this
line
(regex)
--max-args MAX_ARGS maximum number of arguments parsed and displayed,
defaults to
20
examples:
./execsnoop # trace all exec() syscalls
./execsnoop -x # include failed exec()s
./execsnoop -t # include timestamps
./execsnoop -q # add
"quotemarks"
around arguments
./execsnoop -n main # only print command lines containing
"main"
./execsnoop -l tpkg # only print command where arguments contains
"tpkg"
[root
@yang
-
01
tools]# execsnoop
PCOMM PID PPID RET ARGS
bash
17436
1642
0
/usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_5690.service
bash
17437
1642
0
systemctl
17437
1642
0
/usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_3307.service /usr/bin/systemctl is-active mysqld_3307.service
systemctl
17436
1642
0
/usr/bin/systemctl is-active mysqld_5690.service
bash
17439
1642
0
/usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_3307.service
systemctl
17439
1642
0
/usr/bin/systemctl is-active mysqld_3307.service
bash
17440
1642
0
/usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_5690.service
systemctl
17440
1642
0
/usr/bin/systemctl is-active mysqld_5690.service
bash
17442
1642
0
/usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_3307.service
systemctl
17442
1642
0
/usr/bin/systemctl is-active mysqld_3307.service
bash
17443
1642
0
/usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_5690.service
systemctl
17443
1642
0
/usr/bin/systemctl is-active mysqld_5690.service
bash
17445
1642
0
/usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_3307.service
bash
17446
1642
0
/usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_5690.service
##################################################################################################################################################################
# opensnoop 通过跟踪 open() 系统调用来工作。opensnoop 的一些优点在于它能在系统范围内工作，并且跟踪所有进程的 open() 系统调用。
[root
@yang
-
01
tools]# opensnoop -h
usage: opensnoop [-h] [-T] [-U] [-x] [-p PID] [-t TID] [-u UID] [-d DURATION]
[-n NAME] [-e] [-f FLAG_FILTER]
Trace open() syscalls
optional arguments:
-h, --help show
this
help message and exit
-T, --timestamp include timestamp on output
-U, --print-uid print UID column
-x, --failed only show failed opens
-p PID, --pid PID trace
this
PID only
-t TID, --tid TID trace
this
TID only
-u UID, --uid UID trace
this
UID only
-d DURATION, --duration DURATION
total duration of trace in seconds
-n NAME, --name NAME only print process names containing
this
name
-e, --extended_fields
show extended fields
-f FLAG_FILTER, --flag_filter FLAG_FILTER
filter on flags argument (e.g., O_WRONLY)
examples:
./opensnoop # trace all open() syscalls
./opensnoop -T # include timestamps
./opensnoop -U # include UID
./opensnoop -x # only show failed opens
./opensnoop -p
181
# only trace PID
181
./opensnoop -t
123
# only trace TID
123
./opensnoop -u
1000
# only trace UID
1000
./opensnoop -d
10
# trace
for
10
seconds only
./opensnoop -n main # only print process names containing
"main"
./opensnoop -e # show extended fields
./opensnoop -f O_WRONLY -f O_RDWR # only print calls
for
writing
[root
@yang
-
01
tools]# opensnoop
PID COMM FD ERR PATH
1642
ustats
14
0
./logs
18699
opensnoop -
1
2
/usr/lib64/python2.
7
/encodings/ascii.so
18699
opensnoop -
1
2
/usr/lib64/python2.
7
/encodings/asciimodule.so
18699
opensnoop
16
0
/usr/lib64/python2.
7
/encodings/ascii.py
18699
opensnoop
17
0
/usr/lib64/python2.
7
/encodings/ascii.pyc
1309
uagent
16
0
/action-dmp/ustats/ustats.pid
1309
uagent
16
0
/proc/
1642
/cmdline
1309
uagent
16
0
/action-dmp/umon-mgr/umon-mgr.pid
1309
uagent
16
0
/proc/
1684
/cmdline
1309
uagent
16
0
/action-dmp/urman-agent/urman-agent.pid
1309
uagent
16
0
/proc/
1672
/cmdline
1309
uagent
16
0
/action-dmp/ucore/ucore.pid
1309
uagent
16
0
/proc/
1538
/cmdline
1309
uagent
16
0
/action-dmp/uguard-agent/uguard-agent.pid
1309
uagent
16
0
/proc/
1664
/cmdline
1309
uagent
16
0
/action-dmp/uguard-mgr/uguard-mgr.pid
1309
uagent
16
0
/proc/
1636
/cmdline
1309
uagent
16
0
/action-dmp/umon/umon.pid
1309
uagent
16
0
/proc/
1734
/cmdline
1309
uagent
16
0
/action-dmp/udeploy/udeploy.pid
1309
uagent
16
0
/proc/
1644
/cmdline
1309
uagent
16
0
/action-dmp/umc/umc.pid
1309
uagent
16
0
/proc/
1638
/cmdline
1309
uagent
16
0
/action-dmp/uproxy/uproxy.pid
1309
uagent
16
0
/proc/
22496
/cmdline
1309
uagent
16
0
/action-dmp/urman-mgr/urman-mgr.pid
1309
uagent
16
0
/proc/
1640
/cmdline
1664
uguard-agent
18
0
./logs
1642
ustats
14
0
/dev/
null
1642
ustats
40
0
/dev/
null
18703
bash
3
0
/etc/ld.so.cache
18703
bash
3
0
/lib64/libtinfo.so.
5
18703
bash
3
0
/lib64/libdl.so.
2
18703
bash
3
0
/lib64/libc.so.
6
18703
bash -
1
6
/dev/tty
18703
bash
3
0
/usr/lib/locale/locale-archive
18703
bash
3
0
/proc/meminfo
18703
bash
3
0
/usr/lib64/gconv/gconv-modules.cache
18703
bash
3
0
/etc/nsswitch.conf
18703
bash
3
0
/etc/ld.so.cache
18703
bash
3
0
/lib64/libnss_files.so.
2
18703
bash
3
0
/etc/passwd
5448
mysqld -
1
2
./universe/u_delay.frm
18703
systemctl
3
0
/etc/ld.so.cache
18703
systemctl
3
0
/lib64/librt.so.
1
18703
systemctl
3
0
/lib64/libselinux.so.
1
18703
systemctl
3
0
/lib64/liblzma.so.
5
18703
systemctl
3
0
/lib64/liblz4.so.
1
18703
systemctl
3
0
/lib64/libgcrypt.so.
11
18703
systemctl
3
0
/lib64/libcap.so.
2
18703
systemctl
3
0
/lib64/libgcc_s.so.
1
18703
systemctl
3
0
/lib64/libpthread.so.
0
18703
systemctl
3
0
/lib64/libc.so.
6
18703
systemctl
3
0
/lib64/libpcre.so.
1
18703
systemctl
3
0
/lib64/libdl.so.
2
18703
systemctl
3
0
/lib64/libgpg-error.so.
0
18703
systemctl
3
0
/lib64/libattr.so.
1
18704
bash
3
0
/etc/ld.so.cache
18704
bash
3
0
/lib64/libtinfo.so.
5
##################################################################################################################################################################
# xfsslower 工具可以跟踪大于
1
毫秒（参数）延迟的常见 XFS 文件系统操作。
# 在 xfsslower 工具运行的时候，通过在 XFS 中动态地检测内核函数实现的，并当它结束的时候解除该检测。
# 这个 bcc 工具也有其它文件系统的版本：ext4slower、btrfsslower、zfsslower 和 nfsslower。
[root
@yang
-
01
tools]# xfsslower -h
usage: xfsslower [-h] [-j] [-p PID] [min_ms]
Trace common XFS file operations slower than a threshold
positional arguments:
min_ms minimum I/O duration to trace, in ms (
default
10
)
optional arguments:
-h, --help show
this
help message and exit
-j, --csv just print fields: comma-separated values
-p PID, --pid PID trace
this
PID only
examples:
./xfsslower # trace operations slower than
10
ms (
default
)
./xfsslower
1
# trace operations slower than
1
ms
./xfsslower -j
1
# ...
1
ms, parsable output (csv)
./xfsslower
0
# trace all operations (warning: verbose)
./xfsslower -p
185
# trace PID
185
only
[root
@yang
-
01
tools]# xfsslower
1
Tracing XFS operations slower than
1
ms
TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME
17
:
02
:
44
umon-mgr
1684
S
0
0
5.93
detail.log
17
:
02
:
44
umc
1638
S
0
0
1.05
detail.log
17
:
02
:
45
mysqld
10763
W
81920
0
8.12
temp_16.ibt
17
:
02
:
45
mysqld
10763
W
16384
80
2.16
temp_16.ibt
17
:
02
:
45
ustats
1642
S
0
0
3.12
brief.log
17
:
02
:
46
mysqld
10763
W
835584
1024
7.81
ibdata1
17
:
02
:
46
mysqld
10763
S
0
0
31.14
ib_logfile0
17
:
02
:
46
ucore
1538
S
0
0
5.89
raft.db
17
:
02
:
46
ustats
1642
S
0
0
4.04
detail.log
17
:
02
:
46
mysqld
10763
S
0
0
3.58
ib_logfile0
17
:
02
:
46
ustats
1642
S
0
0
1.49
brief.log
17
:
02
:
46
umc
1638
S
0
0
13.70
brief.log
17
:
02
:
46
uguard-mgr
1636
S
0
0
9.62
detail.log
17
:
02
:
46
ucore
1538
S
0
0
8.87
detail.log
17
:
02
:
48
ustats
1642
S
0
0
2.16
brief.log
17
:
02
:
49
ustats
1642
S
0
0
1.41
brief.log
17
:
02
:
49
uguard-agent
1664
S
0
0
26.30
brief.log
17
:
02
:
49
uguard-agent
1664
S
0
0
6.56
detail.log
17
:
02
:
49
mysqld
10763
S
0
0
1.89
ib_logfile0
17
:
02
:
50
ustats
1642
S
0
0
1.01
brief.log
17
:
02
:
50
mysqld
10763
S
0
0
1.18
ib_logfile0
##################################################################################################################################################################
# 可以用来实现高效的内核摘要统计。从内核层到用户层的数据传输仅仅是“计数”列。 用户级程序生成其余的。
[root
@yang
-
01
tools]# biolatency -h
usage: biolatency [-h] [-T] [-Q] [-m] [-D] [-F] [interval] [count]
Summarize block device I/O latency as a histogram
positional arguments:
interval output interval, in seconds
count number of outputs
optional arguments:
-h, --help show
this
help message and exit
-T, --timestamp include timestamp on output
-Q, --queued include OS queued time in I/O time
-m, --milliseconds millisecond histogram
-D, --disks print a histogram per disk device
-F, --flags print a histogram per set of I/O flags
examples:
./biolatency # summarize block I/O latency as a histogram
./biolatency
1
10
# print
1
second summaries,
10
times
./biolatency -mT
1
# 1s summaries, milliseconds, and timestamps
./biolatency -Q # include OS queued time in I/O time
./biolatency -D # show each disk device separately
./biolatency -F # show I/O flags separately
[root
@yang
-
01
tools]# biolatency
Tracing block device I/O... Hit Ctrl-C to end.
^C
usecs : count distribution
0
->
1
:
0
| |
2
->
3
:
0
| |
4
->
7
:
0
| |
8
->
15
:
0
| |
16
->
31
:
13
|* |
32
->
63
:
306
|********************************* |
64
->
127
:
367
|****************************************|
128
->
255
:
254
|*************************** |
256
->
511
:
12
|* |
512
->
1023
:
1
| |
1024
->
2047
:
1
| |
2048
->
4095
:
9
| |
4096
->
8191
:
10
|* |
8192
->
16383
:
9
| |
16384
->
32767
:
3
| |
32768
->
65535
:
2
| |
##################################################################################################################################################################
# tcplife显示 TCP 会话的生命周期和吞吐量统计
[root
@yang
-
01
tools]# tcplife -h
usage: tcplife [-h] [-T] [-t] [-w] [-s] [-p PID] [-L LOCALPORT]
[-D REMOTEPORT]
Trace the lifespan of TCP sessions and summarize
optional arguments:
-h, --help show
this
help message and exit
-T, --time include time column on output (HH:MM:SS)
-t, --timestamp include timestamp on output (seconds)
-w, --wide wide column output (fits IPv6 addresses)
-s, --csv comma separated values output
-p PID, --pid PID trace
this
PID only
-L LOCALPORT, --localport LOCALPORT
comma-separated list of local ports to trace.
-D REMOTEPORT, --remoteport REMOTEPORT
comma-separated list of remote ports to trace.
examples:
./tcplife # trace all TCP connect()s
./tcplife -t # include time column (HH:MM:SS)
./tcplife -w # wider colums (fit IPv6)
./tcplife -stT # csv output, with times & timestamps
./tcplife -p
181
# only trace PID
181
./tcplife -L
80
# only trace local port
80
./tcplife -L
80
,
81
# only trace local ports
80
and
81
./tcplife -D
80
# only trace remote port
80
[root
@yang
-
01
tools]# tcplife
PID COMM LADDR LPORT RADDR RPORT TX_KB RX_KB MS
854
jsvc ::ffff:
127.0
.
0.1
33222
::ffff:
127.0
.
0.1
3307
0
0
1.91
10763
mysqld ::ffff:
127.0
.
0.1
3307
::ffff:
127.0
.
0.1
33222
0
0
1.91
854
jsvc ::ffff:
127.0
.
0.1
33224
::ffff:
127.0
.
0.1
3307
0
0
2.02
10763
mysqld ::ffff:
127.0
.
0.1
3307
::ffff:
127.0
.
0.1
33224
0
0
2.03
1638
umc
10.186
.
60.24
48697
10.186
.
60.24
5704
0
0
3.01
1636
uguard-mgr ::ffff:
10.186
.
60.24
5704
::ffff:
10.186
.
60.24
48697
0
0
3.03
854
jsvc ::ffff:
127.0
.
0.1
33228
::ffff:
127.0
.
0.1
3307
0
0
1.76
10763
mysqld ::ffff:
127.0
.
0.1
3307
::ffff:
127.0
.
0.1
33228
0
0
1.77
1684
umon-mgr ::
1
37504
::
1
8200
0
0
0.03
1684
umon-mgr
127.0
.
0.1
48056
127.0
.
0.1
8200
0
0
0.02
854
jsvc ::ffff:
127.0
.
0.1
33234
::ffff:
127.0
.
0.1
3307
0
0
2.02
10763
mysqld ::ffff:
127.0
.
0.1
3307
::ffff:
127.0
.
0.1
33234
0
0
2.04
1642
ustats
127.0
.
0.1
33236
127.0
.
0.1
3307
0
0
3.13
10763
mysqld ::ffff:
127.0
.
0.1
3307
::ffff:
127.0
.
0.1
33236
0
0
3.15
1642
ustats
127.0
.
0.1
45738
127.0
.
0.1
5690
0
0
3.33
5448
mysqld ::ffff:
127.0
.
0.1
5690
::ffff:
127.0
.
0.1
45738
0
0
3.35
854
jsvc ::ffff:
127.0
.
0.1
33240
::ffff:
127.0
.
0.1
3307
0
0
1.65
10763
mysqld ::ffff:
127.0
.
0.1
3307
::ffff:
127.0
.
0.1
33240
0
0
1.66
####################################################################################################################################################################
# 检测用于名称解析的 gethostbyname(
3
) 和相关的库调用
[root
@yang
-
01
tools]# gethostlatency -h
usage: gethostlatency [-h] [-p PID]
Show latency
for
getaddrinfo/gethostbyname[
2
] calls
optional arguments:
-h, --help show
this
help message and exit
-p PID, --pid PID trace
this
PID only
examples:
./gethostlatency # trace all TCP accept()s
./gethostlatency -p
181
# only trace PID
181
[root
@yang
-
01
tools]# gethostlatency
TIME PID COMM LATms HOST
17
:
37
:
55
903
jsvc
0.28
localhost
17
:
38
:
03
22079
umon
0.28
localhost
17
:
38
:
08
14308
umon-mgr
0.23
localhost
17
:
38
:
16
919
jsvc
0.17
yang-
01
17
:
38
:
25
903
jsvc
0.28
localhost
17
:
38
:
37
24347
umon
0.25
localhost
17
:
38
:
38
1780
umon-mgr
0.34
localhost
17
:
38
:
38
22085
umon
0.22
localhost
17
:
38
:
43
1780
umon-mgr
0.22
localhost
17
:
38
:
45
893
jsvc
0.08
yang-
01
#####################################################################################################################################################################
# trace 工具由 Sasha Goldshtein 提供，并提供了一些基本的 printf(
1
) 功能和自定义探针。
[root
@yang
-
01
tools]# trace -h
usage: trace [-h] [-b BUFFER_PAGES] [-p PID] [-L TID] [-v] [-Z STRING_SIZE]
[-S] [-M MAX_EVENTS] [-t] [-T] [-C] [-B] [-s SYM_FILE_LIST] [-K]
[-U] [-a] [-I header]
probe [probe ...]
Attach to functions and print trace messages.
positional arguments:
probe probe specifier (see examples)
optional arguments:
-h, --help show
this
help message and exit
-b BUFFER_PAGES, --buffer-pages BUFFER_PAGES
number of pages to use
for
perf_events ring buffer
(
default
:
64
)
-p PID, --pid PID id of the process to trace (optional)
-L TID, --tid TID id of the thread to trace (optional)
-v, --verbose print resulting BPF program code before executing
-Z STRING_SIZE, --string-size STRING_SIZE
maximum size to read from strings
-S, --include-self
do
not filter trace's own pid from the trace
-M MAX_EVENTS, --max-events MAX_EVENTS
number of events to print before quitting
-t, --timestamp print timestamp column (offset from trace start)
-T, --time print time column
-C, --print_cpu print CPU id
-B, --bin_cmp allow to use STRCMP with binary values
-s SYM_FILE_LIST, --sym_file_list SYM_FILE_LIST
coma separated list of symbol files to use
for
symbol
resolution
-K, --kernel-stack output kernel stack trace
-U, --user-stack output user stack trace
-a, --address print virtual address in stacks
-I header, --include header
additional header files to include in the BPF program
as either full path, or relative to current working
directory, or relative to
default
kernel header search
path
EXAMPLES:
trace do_sys_open
Trace the open syscall and print a
default
trace message when entered
trace
'do_sys_open "%s", arg2'
Trace the open syscall and print the filename being opened
trace
'sys_read (arg3 > 20000) "read %d bytes", arg3'
Trace the read syscall and print a message
for
reads >
20000
bytes
trace
'r::do_sys_open "%llx", retval'
Trace the
return
from the open syscall and print the
return
value
trace
'c:open (arg2 == 42) "%s %d", arg1, arg2'
Trace the open() call from libc only
if
the flags (arg2) argument is
42
trace
'c:malloc "size = %d", arg1'
Trace malloc calls and print the size being allocated
trace
'p:c:write (arg1 == 1) "writing %d bytes to STDOUT", arg3'
Trace the write() call from libc to monitor writes to STDOUT
trace
'r::__kmalloc (retval == 0) "kmalloc failed!"'
Trace returns from __kmalloc which returned a
null
pointer
trace
'r:c:malloc (retval) "allocated = %x", retval'
Trace returns from malloc and print non-NULL allocated buffers
trace
't:block:block_rq_complete "sectors=%d", args->nr_sector'
Trace the block_rq_complete kernel tracepoint and print # of tx sectors
trace
'u:pthread:pthread_create (arg4 != 0)'
Trace the USDT probe pthread_create when its 4th argument is non-zero
trace
'p::SyS_nanosleep(struct timespec *ts) "sleep for %lld ns", ts->tv_nsec'
Trace the nanosleep syscall and print the sleep duration in ns
trace -I
'linux/fs.h'
\
'p::uprobe_register(struct inode *inode) "a_ops = %llx", inode->i_mapping->a_ops'
Trace the uprobe_register inode mapping ops, and the symbol can be found
in /proc/kallsyms
trace -I
'kernel/sched/sched.h'
\
'p::__account_cfs_rq_runtime(struct cfs_rq *cfs_rq) "%d", cfs_rq->runtime_remaining'
Trace the cfs scheduling runqueue remaining runtime. The struct cfs_rq is defined
in kernel/sched/sched.h which is in kernel source tree and not in kernel-devel
package
. So
this
command needs to run at the kernel source tree root directory
so that the added header file can be found by the compiler.
trace -I
'net/sock.h'
\
'udpv6_sendmsg(struct sock *sk) (sk->sk_dport == 13568)'
Trace udpv6 sendmsg calls only
if
socket's destination port is equal
to
53
(DNS;
13568
in big endian order)
trace -I
'linux/fs_struct.h'
'mntns_install "users = %d", $task->fs->users'
Trace the number of users accessing the file system of the current task
|
内核版本低的限制
# bcc 基于 eBPF 开发（需要 Linux
3.15
及更高版本）。bcc 使用的大部分内容都需要 Linux
4.1
及更高版本。
[root
@yang
-
01
tools]# dbstat mysql -p
10763
-u
Traceback (most recent call last):
File
"/usr/share/bcc/tools/dbstat"
, line
88
, in <module>
usdt.enable_probe(
"query__start"
,
"probe_start"
)
File
"/usr/lib/python2.7/site-packages/bcc/usdt.py"
, line
154
, in enable_probe
probe
bcc.usdt.USDTException: failed to enable probe
'query__start'
; a possible cause can be that the probe requires a pid to enable
[root
@yang
-
01
tools]# dbslower mysql -p
10763
-m
30
Traceback (most recent call last):
File
"/usr/share/bcc/tools/dbslower"
, line
193
, in <module>
usdt.enable_probe(
"query__start"
,
"query_start"
)
File
"/usr/lib/python2.7/site-packages/bcc/usdt.py"
, line
154
, in enable_probe
probe
bcc.usdt.USDTException: failed to enable probe
'query__start'
; a possible cause can be that the probe requires a pid to enable
# OS内核版本, >=
4.4
存在统计Bug, 部分功能需要>=
4.13
# "bcc.usdt.USDTException: failed to enable probe
'query__start'
; a possible cause can be that the probe requires a pid to enable” 需要有Dtrace tracepoint的MySQL, 需要重新编译. https:
//dev.mysql.com/doc/refman/5.7/en/dba-dtrace-server.html
# 使用bcc需要root权限
|
[root@yang-01 tools]
# biolatency -D 2
Tracing block device I
/O
... Hit Ctrl-C to end.
disk =
'vda'
usecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 0 | |
16 -> 31 : 0 | |
32 -> 63 : 63 |*********************************** |
64 -> 127 : 71 |****************************************|
128 -> 255 : 46 |************************* |
256 -> 511 : 4 |** |
512 -> 1023 : 0 | |
1024 -> 2047 : 0 | |
2048 -> 4095 : 3 |* |
4096 -> 8191 : 3 |* |
disk =
''
usecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 0 | |
16 -> 31 : 0 | |
32 -> 63 : 0 | |
64 -> 127 : 0 | |
128 -> 255 : 0 | |
256 -> 511 : 0 | |
512 -> 1023 : 0 | |
1024 -> 2047 : 0 | |
2048 -> 4095 : 1 |****************************************|
disk =
'vda'
usecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 0 | |
16 -> 31 : 10 |*** |
32 -> 63 : 84 |******************************** |
64 -> 127 : 103 |****************************************|
128 -> 255 : 43 |**************** |
256 -> 511 : 4 |* |
512 -> 1023 : 0 | |
1024 -> 2047 : 0 | |
2048 -> 4095 : 7 |** |
4096 -> 8191 : 17 |****** |
8192 -> 16383 : 1 | |
16384 -> 32767 : 1 | |
32768 -> 65535 : 1 | |
65536 -> 131071 : 1 | |
131072 -> 262143 : 44 |***************** |
disk =
''
usecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 0 | |
16 -> 31 : 0 | |
32 -> 63 : 0 | |
64 -> 127 : 0 | |
128 -> 255 : 0 | |
256 -> 511 : 0 | |
512 -> 1023 : 0 | |
1024 -> 2047 : 0 | |
2048 -> 4095 : 1 |****************************************|
[root@yang-01 tools]
# ps aux | grep mysql | grep opt
actiont+ 10763 3.3 18.0 4570952 1070628 ? Sl Jul16 1887:06
/opt/mysql/base/8
.0.19
/bin/mysqld
--defaults-
file
=
/opt/mysql/etc/3307/my
.cnf --daemonize --pid-
file
=
/opt/mysql/data/3307/mysqld
.pid --user=actiontech-mysql --socket=
/opt/mysql/data/3307/mysqld
.sock --port=3307
[root@yang-01 tools]
# filetop -p 10763 -C 5
Tracing... Output every 5 secs. Hit Ctrl-C to end
18:01:40 loadavg: 0.14 0.15 0.20 2
/1119
31195
TID COMM READS WRITES R_Kb W_Kb T FILE
10788 mysqld 0 1 0 832 R ibdata1
10909 mysqld 19 0 117 0 R mysql-relay.000044
10794 mysqld 0 99 0 67 R ib_logfile0
10977 mysqld 0 60 0 4 R mysql-relay.000044
10910 mysqld 0 10 0 4 R mysql-bin.000015
10789 mysqld 0 1 0 0 R ib_logfile0
18:01:45 loadavg: 0.21 0.16 0.21 2
/1119
31212
TID COMM READS WRITES R_Kb W_Kb T FILE
10909 mysqld 19 0 116 0 R mysql-relay.000044
10794 mysqld 0 98 0 64 R ib_logfile0
10977 mysqld 0 60 0 4 R mysql-relay.000044
10910 mysqld 0 10 0 4 R mysql-bin.000015
18:01:50 loadavg: 0.19 0.16 0.20 2
/1119
31226
TID COMM READS WRITES R_Kb W_Kb T FILE
10909 mysqld 15 0 88 0 R mysql-relay.000044
10794 mysqld 0 106 0 70 R ib_logfile0
19780 mysqld 2 2 8 7 R ibzMa2hy
10977 mysqld 0 60 0 4 R mysql-relay.000044
10910 mysqld 0 10 0 4 R mysql-bin.000015
19780 mysqld 1 0 0 0 R mysql-bin.000009
19780 mysqld 1 0 0 0 R mysql-bin.000012
19780 mysqld 1 0 0 0 R mysql-bin.000014
19780 mysqld 1 0 0 0 R mysql-bin.000010
19780 mysqld 1 0 0 0 R mysql-bin.000013
19780 mysqld 1 0 0 0 R mysql-bin.000011
[root@yang-01 tools]
# filelife
TIME PID COMM AGE(s) FILE
18:03:26 1823 ustats 0.00 disk-
read
-write-
test
.log
18:03:26 1830 ustats 0.00 disk-
read
-write-
test
.log
18:03:29 1736 umon-mgr 0.01 grafana.db-journal
18:03:41 1823 ustats 0.00 disk-
read
-write-
test
.log
18:03:41 1784 ustats 0.01 disk-
read
-write-
test
.log
|