1. 背景
SQLAdvisor是由美團點評公司技術工程部DBA團隊（北京）開發維護的一個分析SQL給出索引優化建議的工具。它基於MySQL原生態詞法解析，結合分析SQL中的where條件、聚合條件、多表Join關系 給出索引優化建議。目前SQLAdvisor在美團點評內部廣泛應用，公司內部對SQLAdvisor的開發全面轉到github上，開源和內部使用保持一致。
在數據庫運維過程中，優化SQL是業務團隊與DBA團隊的日常任務。例行SQL優化，不僅可以提升程序性能，還能夠降低線上故障的概率。
目前常用的SQL優化方式包括但不限於：業務層優化、SQL邏輯優化、索引優化等。其中索引優化通常通過調整索引或新增索引從而達到SQL優化的目的。索引優化往往可以在短時間內產生非常巨大的效果。如果能夠將索引優化轉化成工具化、標准化的流程，減少人工介入的工作量，無疑會大大提高DBA的工作效率。板面的做法和配料
２. 架構流程圖
3. 環境
* os version
|
1
2
3
4
5
6
7
8
9
10
11
|
[root@SQLAdvisor ~]
# cat /etc/redhat-release
CentOS release 6.8 (Final)
[root@SQLAdvisor ~]
# uname -r
2.6.32-642.3.1.el6.x86_64
[root@SQLAdvisor ~]
# uname -n
SQLAdvisor
[root@SQLAdvisor ~]
# getenforce
Disabled
|
* mysql version
|
1
2
3
4
5
6
7
|
mysql> show variables like
'version'
;
+---------------+--------+
| Variable_name | Value |
+---------------+--------+
| version | 5.7.18 |
+---------------+--------+
1 row
in
set
(0.00 sec)
|
4. 安裝SQLAdvisor
* 獲取最新代碼
|
1
2
3
4
5
6
|
[root@SQLAdvisor ~]
# git clone https://github.com/Meituan-Dianping/SQLAdvisor.git
Initialized empty Git repository
in
/root/SQLAdvisor/
.git/
remote: Counting objects: 1460,
done
.
remote: Total 1460 (delta 0), reused 0 (delta 0), pack-reused 1460
Receiving objects: 100% (1460
/1460
), 19.92 MiB | 209 KiB
/s
,
done
.
Resolving deltas: 100% (368
/368
),
done
.
|
* 安裝依賴項
|
1
2
3
4
5
6
7
|
[root@SQLAdvisor ~]
# yum -y install cmake libaio-devel libffi-devel glib2 glib2-devel
[root@SQLAdvisor ~]
# yum -y install
[root@SQLAdvisor ~]
# yum -y install Percona-Server-shared-56
[root@SQLAdvisor ~]
# ln -s /usr/lib64/libperconaserverclient_r.so.18 /usr/lib64/libperconaserverclient_r.so
|
* 編譯依賴項sqlparser
|
1
2
3
4
5
|
[root@SQLAdvisor ~]
# cd SQLAdvisor/
[root@SQLAdvisor SQLAdvisor]
# cmake -DBUILD_CONFIG=mysql_release -DCMAKE_BUILD_TYPE=debug -DCMAKE_INSTALL_PREFIX=/usr/local/sqlparser ./
[root@SQLAdvisor SQLAdvisor]
# make && make install
|
* 安裝SQLAdvisor
|
1
2
3
4
5
|
[root@SQLAdvisor SQLAdvisor]
# cd sqladvisor/
[root@SQLAdvisor sqladvisor]
# cmake -DCMAKE_BUILD_TYPE=debug ./
[root@SQLAdvisor sqladvisor]
# make
|
* SQLAdvisor Info
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
[root@SQLAdvisor sqladvisor]
# ./sqladvisor --help
Usage:
sqladvisor [OPTION...] sqladvisor
SQL Advisor Summary
Help Options:
-?, --help Show help options
Application Options:
-f, --defaults-
file
sqls
file
-u, --username username
-p, --password password
-P, --port port
-h, --host host
-d, --dbname database name
-q, --sqls sqls
-
v
, --verbose 1:output logs 0:output nothing
|
5. 測試
* 生成測試數據表
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
mysql> create database test1 character
set
utf8mb4;
Query OK, 1 row affected (0.00 sec)
mysql> create table user(
->
id
INT PRIMARY KEY AUTO_INCREMENT,
-> name VARCHAR(64) NOT NULL,
-> age int,
-> sex int
-> )ENGINE=INNODB DEFAULT CHARSET=utf8mb4;
Query OK, 0 rows affected (0.13 sec)
mysql> desc user;
+-------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+-------------+------+-----+---------+----------------+
|
id
| int(11) | NO | PRI | NULL | auto_increment |
| name | varchar(64) | NO | | NULL | |
| age | int(11) | YES | | NULL | |
| sex | int(11) | YES | | NULL | |
+-------+-------------+------+-----+---------+----------------+
4 rows
in
set
(0.01 sec)
|
* 生成測試數據
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
|
mysql> insert into user(name,age, sex)
select
'lisea'
, 25, 1;
Query OK, 1 row affected (0.01 sec)
Records: 1 Duplicates: 0 Warnings: 0
mysql> insert into user(name,age, sex)
select
concat(name,
'1'
), age+1, sex+1 from user;
Query OK, 1 row affected (0.02 sec)
Records: 1 Duplicates: 0 Warnings: 0
mysql> insert into user(name,age, sex)
select
concat(name,
'2'
), age+2, sex from user;
Query OK, 2 rows affected (0.02 sec)
Records: 2 Duplicates: 0 Warnings: 0
mysql> insert into user(name,age, sex)
select
concat(name,
'3'
), age+2, sex from user;
Query OK, 4 rows affected (0.18 sec)
Records: 4 Duplicates: 0 Warnings: 0
.
.
.
.
.
.
mysql> insert into user(name,age, sex)
select
concat(name,
'10'
), age+2, sex from user;
Query OK, 512 rows affected (0.24 sec)
Records: 512 Duplicates: 0 Warnings: 0
mysql> insert into user(name,age, sex)
select
concat(name,
'11'
), age+4, sex from user;
Query OK, 1024 rows affected (0.79 sec)
Records: 1024 Duplicates: 0 Warnings: 0
mysql>
select
count(1) from user;
+----------+
| count(1) |
+----------+
| 2048 |
+----------+
1 row
in
set
(0.01 sec)
|
* 命令行傳參調用測試SQLAdvisor [查找非索引行]
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
|
[root@SQLAdvisor sqladvisor]
# ./sqladvisor -h 127.0.0.1 -P 3306 -u root -p '123' -d test1 -q "select * from user where name = 'lisea'" -v 1
2017-10-27 05:35:49 34059 [Note] 第1步: 對SQL解析優化之后得到的SQL:
select
`*` AS `*` from `test1`.`user` where (`name` =
'lisea'
)
2017-10-27 05:35:49 34059 [Note] 第2步：開始解析where中的條件:(`name` =
'lisea'
)
2017-10-27 05:35:49 34059 [Note] show index from user
2017-10-27 05:35:49 34059 [Note] show table status like
'user'
2017-10-27 05:35:49 34059 [Note]
select
count(*) from (
select
`name` from `user` FORCE INDEX( PRIMARY ) order by
id
DESC limit 1024) `user` where (`name` =
'lisea'
)
2017-10-27 05:35:49 34059 [Note] 第3步：表user的行數:2048,limit行數:1024,得到where條件中(`name` =
'lisea'
)的選擇度:1024
2017-10-27 05:35:49 34059 [Note] 第4步：開始驗證 字段name是不是主鍵。表名:user
2017-10-27 05:35:49 34059 [Note] show index from user where Key_name =
'PRIMARY'
and Column_name =
'name'
and Seq_in_index = 1
2017-10-27 05:35:49 34059 [Note] 第5步：字段name不是主鍵。表名:user
2017-10-27 05:35:49 34059 [Note] 第6步：開始驗證 字段name是不是主鍵。表名:user
2017-10-27 05:35:49 34059 [Note] show index from user where Key_name =
'PRIMARY'
and Column_name =
'name'
and Seq_in_index = 1
2017-10-27 05:35:49 34059 [Note] 第7步：字段name不是主鍵。表名:user
2017-10-27 05:35:49 34059 [Note] 第8步：開始驗證表中是否已存在相關索引。表名:user, 字段名:name, 在索引中的位置:1
2017-10-27 05:35:49 34059 [Note] show index from user where Column_name =
'name'
and Seq_in_index =1
2017-10-27 05:35:49 34059 [Note] 第9步：開始輸出表user索引優化建議:
2017-10-27 05:35:49 34059 [Note] Create_Index_SQL：alter table user add index idx_name(name)
2017-10-27 05:35:49 34059 [Note] 第10步: SQLAdvisor結束!
|
* 命令行傳參調用測試SQLAdvisor [查找索引行]
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
[root@SQLAdvisor sqladvisor]
# ./sqladvisor -h 127.0.0.1 -P 3306 -u root -p '123' -d test1 -q "select * from user where id = 1" -v 1
2017-10-27 05:36:46 34062 [Note] 第1步: 對SQL解析優化之后得到的SQL:
select
`*` AS `*` from `test1`.`user` where (`
id
` = 1)
2017-10-27 05:36:46 34062 [Note] 第2步：開始解析where中的條件:(`
id
` = 1)
2017-10-27 05:36:46 34062 [Note] show index from user
2017-10-27 05:36:46 34062 [Note] show table status like
'user'
2017-10-27 05:36:46 34062 [Note]
select
count(*) from (
select
`
id
` from `user` FORCE INDEX( PRIMARY ) order by
id
DESC limit 1024) `user` where (`
id
` = 1)
2017-10-27 05:36:46 34062 [Note] 第3步：表user的行數:2048,limit行數:1024,得到where條件中(`
id
` = 1)的選擇度:1024
2017-10-27 05:36:46 34062 [Note] 第4步：開始驗證 字段
id
是不是主鍵。表名:user
2017-10-27 05:36:46 34062 [Note] show index from user where Key_name =
'PRIMARY'
and Column_name =
'id'
and Seq_in_index = 1
2017-10-27 05:36:46 34062 [Note] 第5步：字段
id
是主鍵。表名:user
2017-10-27 05:36:46 34062 [Note] 第6步：表user 經過運算得到的索引列首列是主鍵,直接放棄,沒有優化建議
2017-10-27 05:36:46 34062 [Note] 第7步: SQLAdvisor結束!
|
* 配置文件傳參調用
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
|
[root@SQLAdvisor sqladvisor]
# cat sql.cnf
[sqladvisor]
username=root
password=123
host=127.0.0.1
port=3306
dbname=test1
sqls=
select
* from user where name =
'lisea'
[root@SQLAdvisor sqladvisor]
# ./sqladvisor -f sql.cnf -v 1
2017-10-27 05:40:14 34070 [Note] 第1步: 對SQL解析優化之后得到的SQL:
select
`*` AS `*` from `test1`.`user` where (`name` =
'lisea'
)
2017-10-27 05:40:14 34070 [Note] 第2步：開始解析where中的條件:(`name` =
'lisea'
)
2017-10-27 05:40:14 34070 [Note] show index from user
2017-10-27 05:40:14 34070 [Note] show table status like
'user'
2017-10-27 05:40:14 34070 [Note]
select
count(*) from (
select
`name` from `user` FORCE INDEX( PRIMARY ) order by
id
DESC limit 1024) `user` where (`name` =
'lisea'
)
2017-10-27 05:40:14 34070 [Note] 第3步：表user的行數:2048,limit行數:1024,得到where條件中(`name` =
'lisea'
)的選擇度:1024
2017-10-27 05:40:14 34070 [Note] 第4步：開始驗證 字段name是不是主鍵。表名:user
2017-10-27 05:40:14 34070 [Note] show index from user where Key_name =
'PRIMARY'
and Column_name =
'name'
and Seq_in_index = 1
2017-10-27 05:40:14 34070 [Note] 第5步：字段name不是主鍵。表名:user
2017-10-27 05:40:14 34070 [Note] 第6步：開始驗證 字段name是不是主鍵。表名:user
2017-10-27 05:40:14 34070 [Note] show index from user where Key_name =
'PRIMARY'
and Column_name =
'name'
and Seq_in_index = 1
2017-10-27 05:40:14 34070 [Note] 第7步：字段name不是主鍵。表名:user
2017-10-27 05:40:14 34070 [Note] 第8步：開始驗證表中是否已存在相關索引。表名:user, 字段名:name, 在索引中的位置:1
2017-10-27 05:40:14 34070 [Note] show index from user where Column_name =
'name'
and Seq_in_index =1
2017-10-27 05:40:14 34070 [Note] 第9步：開始輸出表user索引優化建議:
2017-10-27 05:40:14 34070 [Note] Create_Index_SQL：alter table user add index idx_name(name)
2017-10-27 05:40:14 34070 [Note] 第10步: SQLAdvisor結束!
|