sql自動審核工具-inception


[inception使用規范及說明文檔](http://mysql-inception.github.io/inception-document/usage/)
[代碼倉庫](https://github.com/mysql-inception/inception)

inception介紹

inception是去哪網團隊開發的一個集審核、執行、備份及生成回滾語句於一身的MySQL自動化運維工具,

可以集成進mysql自動化運維平台來實現sql的自動審核。
開發語言:C/C++ ,在mysql源碼的基礎上改造的

限制

目前只支持通過C/C++接口、Python接口對inception的訪問

inception充當的角色

inception對於自動化應用程序(簡稱rid)來說是服務器,對於數據庫server來說是客戶端。當通過自動化平台提交語句(DML/DDL)后,

執行過程可以概括為(如果rid是Python程序)

  1. 開發人員提交待審核的sql到rid
  2. rid將要訪問的數據庫的連接串封裝到sql語句塊的頭部,然后調用inception
  3. inception對sql進行語法和語義的檢查以及按照參數文件中指定的審核項進行審核
  4. 審核通過后執行語句
  5. 通過解析binlog生成回滾語句保存到參數文件指定的備份庫中

流程圖如下:

inception安裝

可以單獨部署到一台主機上,並在此主機上創建備份庫

yum install gcc gcc-c++ cmake bison openssl-devel ncurses-devel MySQL-pythony
cd /usr/local/src git clone https://github.com/smile-java/inception cd inception # 調用腳本編譯安裝;指定新生成的文件到目錄debug sh inception_build.sh debug

 編譯報錯信息

安裝inception
    sh inception_build.sh debug    
    CMake Error at cmake/bison.cmake:78 (MESSAGE):
      Bison (GNU parser generator) is required to build MySQL.Please install
       bison.
     
     -- Configuring incomplete, errors occurred!
See also "/data0/sql/inception/debug/CMakeFiles/CMakeOutput.log".
See also "/data0/sql/inception/debug/CMakeFiles/CMakeError.log".
make: *** No rule to make target `install'.  Stop.

是缺少依賴包導致,解決是安裝bison包,然后將debug刪除重新編譯安裝即可

啟動inception服務

/data0/sql/inception/debug/sql/Inception --defaults-file=/data0/sql/inception/debug/inc.cnf

注意: 因為Inception支持OSC執行的功能,是通過調用pt-online-schema-change工具來做的,但如果Inception后台啟動(&)的話,可能會導致pt-online-schema-change在執行完成之后,長時間不返回,進而導致Inception卡死的問題,這個問題后面會解決,但現階段請盡量不要使用后台啟動的方式,或者可以使用nohup Inception啟動命令 &的方式來啟動。

inception參數說明

有關審核時參照的規范相關的參數:http://mysql-inception.github.io/inception-document/variables/

有關inception服務器連接的參數

  • port
  • socket=/自己目錄,請自行修改/inc.socket

有關備份庫的參數

  • inception_remote_backup_host //遠程備份庫的host
  • inception_remote_backup_port //遠程備份庫的port
  • inception_remote_system_user //遠程備份庫的一個用戶
  • inception_remote_system_password //上面用戶的密碼

有關支持OSC相關的參數:http://mysql-inception.github.io/inception-document/osc/

使用案例

調用inception的Python模板

#!/usr/bin/python
#-\*-coding: utf-8-\*-
import MySQLdb
sql='/*--user=admin;--password=123123;--host=127.0.0.1;--execute=1;--port=3309;*/\
inception_magic_start;\
use test;\
query #語句塊\
inception_magic_commit;'
try:
    conn=MySQLdb.connect(host='127.0.0.1',user='root',passwd='',db='',port=123123)
    cur=conn.cursor()
    ret=cur.execute(sql)
    result=cur.fetchall()
    num_fields = len(cur.description)
    field_names = [i[0] for i in cur.description]
    print field_names
    for row in result:
        print row[0], "|",row[1],"|",row[2],"|",row[3],"|",row[4],"|",
        row[5],"|",row[6],"|",row[7],"|",row[8],"|",row[9],"|",row[10]
    cur.close()
    conn.close()
except MySQLdb.Error,e:
     print "Mysql Error %d: %s" % (e.args[0], e.args[1])

DDL操作

如果要調用OSC執行,需要開啟參數inception_osc_bin_dir,次參數是會話級別的,每次在提交DDL時可以選擇是否通過OSC執行

#登錄到inception
mysql -uroot -h127.0.0.1 -p123123
# 使用OSC執行ddl
inception set session inception_osc_bin_dir='/usr/local/bin'

query為:create table inctest(id int);

執行結果輸出

 

['ID', 'stage', 'errlevel', 'stagestatus', 'errormessage', 'SQL', 'Affected_rows', 'sequence', 'backup_dbname', 'execute_time', 'sqlsha1']
1 | CHECKED | 0 | Audit completed | None | use test | 0 | '0_0_0' | None | 0 | 
2 | CHECKED | 1 | Audit completed | Set engine to innodb for table 'inctest'.
Set charset to one of 'utf8mb4,utf8' for table 'inctest'.
Set comments for table 'inctest'.
Column 'id' in table 'inctest' have no comments.
Column 'id' in table 'inctest' is not allowed to been nullable.
Set Default value for column 'id' in table 'inctest'
Set a primary key for table 'inctest'. | create table inctest(id int) | 0 | '0_0_1' | 127_0_0_1_3309_test | 0 | 
errormessage列顯示不符合規范的地方,檢查的具體項有
  • 表必須要有主鍵,主鍵為自增,且自增值為1,初始自增值不能大於1
  • 表必須有comment、存儲引擎必須執行為innodb、表字符集必須是inception參數中配置的其中一個
  • 新增的列必須為非空且指定默認值

按規范更改query為:create table inctest(id int unsigned NOT NULL AUTO_INCREMENT comment "id",primary key(id)) ENGINE=InnoDB DEFAULT CHARSET=utf8 comment="test1";

輸出為

['ID', 'stage', 'errlevel', 'stagestatus', 'errormessage', 'SQL', 'Affected_rows', 'sequence', 'backup_dbname', 'execute_time', 'sqlsha1']
1 | RERUN | 0 | Execute Successfully | None | use test | 0 | '1502183472_6520_0' | None | 0.000 | 
2 | EXECUTED | 0 | Execute Successfully
Backup successfully | None | create table inctest(id int unsigned NOT NULL AUTO_INCREMENT comment "id",primary key(id)) ENGINE=InnoDB DEFAULT CHARSET=utf8 comment="test1" | 0 | '1502183472_6520_1' | 127_0_0_1_3309_test | 0.110 | 

如果sqlsha1列輸出非0,則說明使用到了OSC

問題:execute_time列對應的無值

在備份實例中會生成以mysql服務器IP、端口、庫名命名的庫,此處為127_0_0_1_3309_test

09:19:23[127_0_0_1_3309_test](;)> show tables;
+------------------------------------+
| Tables_in_127_0_0_1_3309_test      |
+------------------------------------+
| $_$inception_backup_information$_$ |
| inctest                            |
+------------------------------------+
3 rows in set (0.00 sec)

09:19:26[127_0_0_1_3309_test](;)> select * from $_$inception_backup_information$_$;
+-------------------+-------------------+------------------+------------------+----------------+-----------------------------------------------------------------------------------------------------------------------------------------------+-----------+--------+-----------+------+---------------------+-------------+
| opid_time         | start_binlog_file | start_binlog_pos | end_binlog_file  | end_binlog_pos | sql_statement                                                                                                                                 | host      | dbname | tablename | port | time                | type        |
+-------------------+-------------------+------------------+------------------+----------------+-----------------------------------------------------------------------------------------------------------------------------------------------+-----------+--------+-----------+------+---------------------+-------------+
| 1502183472_6520_1 |                   |                0 |                  |              0 | create table inctest(id int unsigned NOT NULL AUTO_INCREMENT comment "id",primary key(id)) ENGINE=InnoDB DEFAULT CHARSET=utf8 comment="test1" | 127.0.0.1 | test   | inctest   | 3309 | 2017-08-08 17:11:12 | CREATETABLE |
+-------------------+-------------------+------------------+------------------+----------------+-----------------------------------------------------------------------------------------------------------------------------------------------+-----------+--------+-----------+------+---------------------+-------------+
8 rows in set (0.00 sec)

09:19:59[127_0_0_1_3309_test](;)> select * from inctest;
+----+------------------------------+-------------------+
| id | rollback_statement           | opid_time         |
+----+------------------------------+-------------------+
|  2 | DROP TABLE `test`.`inctest`; | 1502183472_6520_1 |
+----+------------------------------+-------------------+
2 rows in set (0.00 sec)

$_$inception_backup_information$_$記錄的是inception的操作日志

回滾sql存儲在和原操作表同名的表中,opid_time是執行語句的唯一序列號,如果知道執行sql的序列號,想要獲得對應的回滾sql,可以執行

select rollback_statement from 127_0_0_1_3309_test.inctest where opid_time =‘1502183472_6520_1’;

 

 

 


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM