一.環境
1.1 jello@jello:~$ uname -a
Linux jello 4.4.0-98-generic #121-Ubuntu SMP Tue Oct 10 14:24:03 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
1.2 jello@jello-Inspiron-N4050:~$ lsb_release -a
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial
二.准備工作.
2.1安裝systemtap (火焰圖依賴於此工具)
sudo apt-get install systemtap
2.2 查找內核對應的debug包
jello@jello$ uname -r
4.4.0-98-generic
那么接下來就是去http://ddebs.ubuntu.com/pool/main/l/linux/下載對應的debug包,找4.4.0-98-generic一致的
2.3 下載對應的debug包
wget http://ddebs.ubuntu.com/pool/main/l/linux/linux-image-4.4.0-98-generic-dbgsym_4.4.0-98.121_amd64.ddeb (這是筆者自己找到的對應下載路徑)
2.4 安裝debug包
sudo dpkg -i linux-image-4.4.0-98-generic-dbgsym_4.4.0-98.121_amd64.ddeb
2.5 安裝nginx
sudo apt-get install nginx
此時在瀏覽器中輸入localhost即可出現以下信息表明安裝ok
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and working. Further configuration is required.
For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.
Thank you for using nginx.
2.5 編寫systemtap腳本nginx.systemtap,內容如下:
global s;
global quit=0;
probe timer.profile {
if (pid() == target()){
if (quit) {
foreach (i in s-) {
print_ustack(i);
printf("\t%d\n",@count(s[i]));
}
exit();
}
else {
s[ubacktrace()] <<< 1;
}
}
}
probe timer.s(20){
quit = 1
}
2.6 使用systemtap
sudo stap --ldd -d /usr/sbin/nginx --all-modules -D MAXMAPENTRIES=256 -D MAXACTION=20000 -D MAXTRACE=100 -D MAXSTRINGLEN=4096 -D MAXBACKTRACE=100 -x 2082 nginx.systemtap --vp 0001 > nginx.out
各參數解析:
--ldd，添加通過ldd解析出來的所有共享庫符號表信息以便為probe到的用戶空間二進制提供信息或者以-d選項列出來，注意：這會使得probe模塊相當的大
-d /usr/sbin/nginx，為給定的模塊(這里是nginx)添加符號表信息到內核對象模塊,這可能使能這些模塊或者程序的象征性traceback，即使他們沒有顯式probe到他們里面
--all-modules,相當於為所有當前被加載的模塊指定'-dkernel'和‘-d'選項
-D MAXMAPENTRIES=256 ,添加給定的c預處理直接到模塊makefile中,這些能被用來限制以下描述的參數,MAXMAPENTRIES=256,設定默認全局數組的最大默認長度為256，通過了解，是用來指定堆棧大小的
-D MAXACTION=20000 設定單個probe hit包含執行的最大語句數目
-D MAXTRACE=100
-D MAXSTRINGLEN=4096 設定字符串最大長度
-D MAXBACKTRACE=100 設定棧幀的最大數目
-x 2082 設定target()的pid為2082
--vp 0001
筆者運行之后出現以下錯誤信息：
jello@jello:~$ sudo stap --ld -d /usr/sbin/nginx --all-modules -D MAXMAPENTRIES=256 -D MAXACTION=20000 -D MAXTRACE=100 -D MAXSTRINGLEN=4096 -D MAXBACKTRACE=100 -x 2082 nginx.systemtap --vp 0001 > nginx.out
In file included from /usr/share/systemtap/runtime/transport/control.c:14:0,
from /usr/share/systemtap/runtime/transport/transport.c:76,
from /usr/share/systemtap/runtime/linux/print.c:17,
from /usr/share/systemtap/runtime/print.c:17,
from /usr/share/systemtap/runtime/runtime_context.h:22,
from /tmp/stapiPADvo/stap_20923465b7e2011fc7e903a669485b1f_9417_src.c:221:
/usr/share/systemtap/runtime/transport/symbols.c: In function ‘_stp_module_update_self’:
/usr/share/systemtap/runtime/transport/symbols.c:243:44: error: ‘struct module’ has no member named ‘symtab’
if (attr->address == (unsigned long) mod->symtab)
^
/usr/share/systemtap/runtime/transport/symbols.c:245:9: error: ‘struct module’ has no member named ‘num_symtab’
mod->num_symtab * sizeof(mod->symtab[0]);
^
/usr/share/systemtap/runtime/transport/symbols.c:245:34: error: ‘struct module’ has no member named ‘symtab’
mod->num_symtab * sizeof(mod->symtab[0]);
^
make[1]: *** [/tmp/stapiPADvo/stap_20923465b7e2011fc7e903a669485b1f_9417_src.o] Error 1
make: *** [_module_/tmp/stapiPADvo] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compiled C into "stap_20923465b7e2011fc7e903a669485b1f_9417.ko" in 9530usr/380sys/12191real ms.
Pass 4: compilation failed. [man error::pass4]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started
從錯誤信息來看是某個mod的某個字段不存在，所以斷定是內核版本過高，但是systemtap版本過低導致的，因此升級systemtap版本
2.6.1 升級systemtap版本
2.6.1.1獲取最新systemtap代碼
方案一.git clone git://sourceware.org/git/systemtap.git (下載速度比較慢,雖然配置翻牆了，仍然慢，推薦方案二)
方案二.tsocks wget https://sourceware.org/systemtap/ftp/releases/systemtap-3.2.tar.gz (tsocks使能wget通過翻牆下載源碼，需要配置vpn進行翻牆)
以上方案均需翻牆，否則下載會非常慢!
2.6.1.2解壓systemtap源碼
tar xvf systemtap-3.2.tar.gz
2.6.1.3 切換到解壓后的systemtap目錄
cd systemtap-3.2
2.6.1.4 進行編譯
2.6.1.4.1 根據README得知：
需要安裝帶有libdwfl庫的elfutils (版本號需要大於或等於0.151)才能正常配合systemtap;
2.6.1.4.2 安裝elfutils
sudo apt-get install elfutils
2.6.1.4.3 驗證elfutils的版本信息
eu-objdump -V
2.6.1.4.4 安裝必不可少的編譯工具
sudo apt-get build-dep systemtap
2.6.1.4.5 創建編譯目錄
mkdir build
2.6.1.4.6 切換到build目錄
cd build
2.6.1.4.7 進行編譯前的配置
../configure
2.6.1.4.8 進行正式編譯
make all
2.6.1.4.8.1 編譯過程中出現以下問題
../bpf-translate.cxx:37:29: fatal error: elfutils/libebl.h: 沒有那個文件或目錄
2.6.1.4.8.2 查看bpf-translate.cxx
#if _ELFUTILS_PREREQ (0, 167)
#include <elfutils/libdwelf.h>
typedef Dwelf_Strent Stap_Strent;
typedef Dwelf_Strtab Stap_Strtab;
#define stap_strtab_init dwelf_strtab_init
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
#define stap_strtab_free dwelf_strtab_free
#define stap_strtab_finalize dwelf_strtab_finalize
#define stap_strent_offset dwelf_strent_off
#else
#include <elfutils/libebl.h>
typedef Ebl_Strent Stap_Strent;
typedef Ebl_Strtab Stap_Strtab;
#define stap_strtab_init ebl_strtabinit
#define stap_strtab_add(X,Y) ebl_strtabadd(X,Y,0)
#define stap_strtab_free ebl_strtabfree
#define stap_strtab_finalize ebl_strtabfinalize
#define stap_strent_offset ebl_strtaboffset
#endif
2.6.1.4.8.2 修改以上內容為:
#if 1
#include <elfutils/libdwelf.h>
typedef Dwelf_Strent Stap_Strent;
typedef Dwelf_Strtab Stap_Strtab;
#define stap_strtab_init dwelf_strtab_init
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
#define stap_strtab_free dwelf_strtab_free
#define stap_strtab_finalize dwelf_strtab_finalize
#define stap_strent_offset dwelf_strent_off
#else
#include <elfutils/libebl.h>
typedef Ebl_Strent Stap_Strent;
typedef Ebl_Strtab Stap_Strtab;
#define stap_strtab_init ebl_strtabinit
#define stap_strtab_add(X,Y) ebl_strtabadd(X,Y,0)
#define stap_strtab_free ebl_strtabfree
#define stap_strtab_finalize ebl_strtabfinalize
#define stap_strent_offset ebl_strtaboffset
#endif
2.6.1.4.8.3 通過以上修改出現以下錯誤信息:
make all-recursive
make[1]: Entering directory '/home/jello/data/development/systemtap-3.2/build'
Making all in .
make[2]: Entering directory '/home/jello/data/development/systemtap-3.2/build'
CXX stap-bpf-translate.o
../bpf-translate.cxx:29:9: error: ‘Dwelf_Strent’ does not name a type
typedef Dwelf_Strent Stap_Strent;
^
../bpf-translate.cxx:30:9: error: ‘Dwelf_Strtab’ does not name a type
typedef Dwelf_Strtab Stap_Strtab;
^
../bpf-translate.cxx:1773:3: error: ‘Stap_Strent’ does not name a type
Stap_Strent *name_ent;
^
../bpf-translate.cxx: In constructor ‘bpf::BPF_Section::BPF_Section(const string&)’:
../bpf-translate.cxx:1782:22: error: class ‘bpf::BPF_Section’ does not have any field named ‘name_ent’
: scn(0), name(n), name_ent(0), data(0), free_data(false)
^
../bpf-translate.cxx: At global scope:
../bpf-translate.cxx:1794:3: error: ‘Stap_Strent’ does not name a type
Stap_Strent *name_ent;
^
../bpf-translate.cxx: In constructor ‘bpf::BPF_Symbol::BPF_Symbol(const string&, bpf::BPF_Section*, long int)’:
../bpf-translate.cxx:1801:14: error: class ‘bpf::BPF_Symbol’ does not have any field named ‘name_ent’
: name(n), name_ent(0)
^
../bpf-translate.cxx: At global scope:
../bpf-translate.cxx:1812:3: error: ‘Stap_Strtab’ does not name a type
Stap_Strtab *str_tab;
^
../bpf-translate.cxx: In constructor ‘bpf::BPF_Output::BPF_Output(int)’:
../bpf-translate.cxx:1827:5: error: class ‘bpf::BPF_Output’ does not have any field named ‘str_tab’
str_tab(stap_strtab_init(true))
^
../bpf-translate.cxx:1827:34: error: ‘dwelf_strtab_init’ was not declared in this scope
str_tab(stap_strtab_init(true))
^
../bpf-translate.cxx: In destructor ‘bpf::BPF_Output::~BPF_Output()’:
../bpf-translate.cxx:1835:20: error: ‘str_tab’ was not declared in this scope
stap_strtab_free(str_tab);
^
../bpf-translate.cxx:1835:27: error: ‘dwelf_strtab_free’ was not declared in this scope
stap_strtab_free(str_tab);
^
../bpf-translate.cxx: In member function ‘bpf::BPF_Section* bpf::BPF_Output::new_scn(const string&)’:
../bpf-translate.cxx:1854:6: error: ‘struct bpf::BPF_Section’ has no member named ‘name_ent’
n->name_ent = stap_strtab_add(str_tab, n->name.c_str());
^
../bpf-translate.cxx:1854:33: error: ‘str_tab’ was not declared in this scope
n->name_ent = stap_strtab_add(str_tab, n->name.c_str());
^
../bpf-translate.cxx:32:48: note: in definition of macro ‘stap_strtab_add’
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
^
../bpf-translate.cxx:32:51: error: ‘dwelf_strtab_add’ was not declared in this scope
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
^
../bpf-translate.cxx:1854:17: note: in expansion of macro ‘stap_strtab_add’
n->name_ent = stap_strtab_add(str_tab, n->name.c_str());
^
../bpf-translate.cxx: In member function ‘bpf::BPF_Symbol* bpf::BPF_Output::new_sym(const string&, bpf::BPF_Section*, long int)’:
../bpf-translate.cxx:1864:6: error: ‘struct bpf::BPF_Symbol’ has no member named ‘name_ent’
s->name_ent = stap_strtab_add(str_tab, s->name.c_str());
^
../bpf-translate.cxx:1864:33: error: ‘str_tab’ was not declared in this scope
s->name_ent = stap_strtab_add(str_tab, s->name.c_str());
^
../bpf-translate.cxx:32:48: note: in definition of macro ‘stap_strtab_add’
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
^
../bpf-translate.cxx:32:51: error: ‘dwelf_strtab_add’ was not declared in this scope
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
^
../bpf-translate.cxx:1864:17: note: in expansion of macro ‘stap_strtab_add’
s->name_ent = stap_strtab_add(str_tab, s->name.c_str());
^
../bpf-translate.cxx: In function ‘void bpf::output_symbols_sections(bpf::BPF_Output&)’:
../bpf-translate.cxx:2183:31: error: ‘struct bpf::BPF_Output’ has no member named ‘str_tab’
stap_strtab_finalize(eo.str_tab, str->data);
^
../bpf-translate.cxx:2183:49: error: ‘dwelf_strtab_finalize’ was not declared in this scope
stap_strtab_finalize(eo.str_tab, str->data);
^
../bpf-translate.cxx:2190:39: error: ‘struct bpf::BPF_Symbol’ has no member named ‘name_ent’
b->st_name = stap_strent_offset(s->name_ent);
^
../bpf-translate.cxx:2190:47: error: ‘dwelf_strent_off’ was not declared in this scope
b->st_name = stap_strent_offset(s->name_ent);
^
../bpf-translate.cxx:2196:29: error: ‘struct bpf::BPF_Output’ has no member named ‘str_tab’
stap_strtab_finalize(eo.str_tab, str->data);
^
../bpf-translate.cxx:2196:47: error: ‘dwelf_strtab_finalize’ was not declared in this scope
stap_strtab_finalize(eo.str_tab, str->data);
^
../bpf-translate.cxx:2203:48: error: ‘struct bpf::BPF_Section’ has no member named ‘name_ent’
s->shdr->sh_name = stap_strent_offset(s->name_ent);
^
../bpf-translate.cxx:2203:56: error: ‘dwelf_strent_off’ was not declared in this scope
s->shdr->sh_name = stap_strent_offset(s->name_ent);
從以上信息可知：以上修改反而會引入更多的問題,因此看一下代碼
2.6.1.4.8.4 將sytemtap的代碼還原
2.6.1.4.8.5 清掉build目錄
rm -rf build
2.6.1.4.8.6 重新創建build目錄
mkdir build
2.6.1.4.8.7 切換到build目錄
cd build
2.6.1.4.8.8 查找elfutils目錄
sudo find / |grep elfutils
筆者找到elfutils對應的頭文件路徑是/usr/include (默認使用apt-get install安裝的軟件對應的頭文件在/usr/include目錄下)
2.6.1.4.8.9 指定elfutils的頭文件路徑進行配置並且指定安裝目錄為/opt/
rm * -rf (需要清一下之前在build目錄下生成的文件)
../configure --with-elfutils=/usr/include --prefix=/opt/
出現以下錯誤信息:
checking for libvirt... no
checking for libxml2... yes
configure: WARNING: will not build systemtap virt support, cannot find libvirt headers
checking for python-config... /usr/bin/python-config
checking Python.h usability... yes
checking Python.h presence... yes
checking for Python.h... yes
checking for python3-config... /usr/bin/python3-config
checking Python.h usability... yes
checking Python.h presence... yes
checking for Python.h... yes
checking for jsonc... no
checking for ncurses... yes
checking for assembler .section "?" flags support... yes
checking linux/bpf.h usability... yes
checking linux/bpf.h presence... yes
checking for linux/bpf.h... yes
configure: error: No /usr/include/configure, forgot to run autoreconf -i?
從此信息中發現缺少configure，應該與elfutils有關,那么下載與當前elfutils對應的版本源代碼
2.6.1.4.8.10 查看當前elfutils版本的方法:
eu-objdump -V
objdump (elfutils) 0.165
Copyright (C) 2012 Red Hat, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Ulrich Drepper.
版本為0.165
2.6.1.4.8.11 獲取elfutils的源碼
筆者找到的0.165下載地址為:https://sourceware.org/elfutils/ftp/0.165/elfutils-0.165.tar.bz2
2.6.1.4.8.12 下載elfutils源碼
tsocks wget https://sourceware.org/elfutils/ftp/0.165/elfutils-0.165.tar.bz2 (翻牆下載才快)
2.6.1.4.8.13解壓elfutils
tar xvf elfutils-0.165.tar.bz2
2.6.1.4.8.14 指定elfutils源碼的路徑(筆者的elfutils源碼路徑在/home/jello/elfutils-0.165下)並指定systemtap的安裝路徑
../configure --with-elfutils=/home/jello/elfutils-0.165 --prefix=/opt/
2.6.1.4.8.15 開始編譯
make
2.6.1.4.8.16 編譯又出現錯誤
systemtap-3.2/build/python/../includes/sys -fPIC -I/usr/include/python2.7 -c HelperSDT/_HelperSDT.c -o /home/jello/systemtap-3.2/build/python/py2build/temp.linux-x86_64-2.7/HelperSDT/_HelperSDT.o
HelperSDT/_HelperSDT.c:10:21: fatal error: sys/sdt.h: 沒有那個文件或目錄
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Makefile:621: recipe for target 'all-local' failed
make[2]: *** [all-local] Error 1
make[2]: Leaving directory '/home/jello/systemtap-3.2/build/python'
Makefile:2012: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/jello/systemtap-3.2/build'
Makefile:717: recipe for target 'all' failed
make: *** [all] Error 2
從以上信息可知:沒有頭文件sys/sdt.h,那么就查找sys/sdt.h的路徑
2.6.1.4.8.17 查找sys/sdt.h的路徑
sudo find /|grep "sys/sdt.h"
發現sys/sdt.h在systemtap-3.2的includes目錄下
2.6.1.4.8.18 修改build/python目錄下的Makefile
修改前的內容:
AM_CPPFLAGS = -I$(srcdir)/../includes \
-I$(abs_builddir)/../includes/sys修改后的內容:AM_CPPFLAGS = -I$(srcdir)/../includes \
-I$(abs_builddir)/../includes/sys \ -I$(abs_srcdir)/../includes
2.6.1.4.8.19 再次編譯又出現以下錯誤(cd ../../python; CFLAGS="-I../../python/../includes -I/home/jello/systemtap-3.2/build/python/../includes/sys -I/home/jello/systemtap-3.2/build/../python/../includes" /usr/bin/python3 setup.py build \
--build-base /home/jello/systemtap-3.2/build/python/py3build \
--verbose)
Traceback (most recent call last):
File "setup.py", line 9, in <module>
from setuptools import setup, Extension
ImportError: No module named 'setuptools'
Makefile:622: recipe for target 'all-local' failed
make[2]: *** [all-local] Error 1
make[2]: Leaving directory '/home/jello/data/development/systemtap-3.2/build/python'
Makefile:2012: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/jello/data/development/systemtap-3.2/build'
Makefile:717: recipe for target 'all' failed
make: *** [all] Error 2
從以上信息可知:python缺少setuptools模塊,所以安裝setuptools模塊,並且使用python3進行安裝
2.6.1.4.8.20 安裝setuptools模塊
2.6.1.4.8.20.1 獲取模塊
wget https://bootstrap.pypa.io/ez_setup.py
2.6.1.4.8.20.2 安裝模塊
sudo python3 ez_setup.py2.6.1.4.8.21 繼續編譯
make一切ok2.6.1.4.9 安裝systemtapsudo make install2.6.1.4.10 卸載原來使用apt-get install安裝的systemtapsudo apt-get remove systemtap2.6.1.4.11 設置環境變量使新安裝的stap生效
vi /etc/profile往文件最后加入一句:export "PATH=/opt/bin:$PATH"2.6.1.5 驗證systap是否安裝成功stap -VSystemtap translator/driver (version 3.2/0.165, non-git sources)
Copyright (C) 2005-2017 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.18 ... 4.11
enabled features: AVAHI PYTHON2 PYTHON3 LIBSQLITE3 LIBXML2 NLS NSS READLINE以上信息中發現版本為3.2/0.165，表明systemtap已經安裝成功
2.6.2 啟動stap
2.6.2.1 先切換為最高權限
sudo su -l
2.6.2.2 啟動stap (之前寫過一個nginx.systemtap文件，記得指定該文件路徑，筆者將nginx.systemtap放在了當前目錄下，因此直接的指定文件）
stap --ld -d /usr/sbin/nginx --all-modules -D MAXMAPENTRIES=256 -D MAXACTION=20000 -D MAXTRACE=100 -D MAXSTRINGLEN=4096 -D MAXBACKTRACE=100 -x 2082 nginx.systemtap --vp 0001 > nginx.out
2.7重開終端進行壓力測試(可選)
進行壓力測試:
ab -n 900000 -c 50 http://192.168.2.323/index.php /*192.168.2.323為筆者的本地ip地址，注意修改成自己的本地ip地址*/
2.8 FlameFraph
2.8.1 獲取FlameGraph
git clone https://github.com/brendangregg/FlameGraph.git
2.8.2
由於2.6.2.2一直未生成nginx.out,因此需要找一下nginx.systemtap是否有錯誤
未寫完，待續...