mysqlbinlog
MySQL官方原生提供的解析(binlog)二進制日志的工具
用法
mysqlbinlog --no-defaults --help
Usage: mysqlbinlog [options] log-files
-?, --help Display this help and exit.
--base64-output=name
Determine when the output statements should be
base64-encoded BINLOG statements: 'never' disables it and
works only for binlogs without row-based events;
'decode-rows' decodes row events into commented
pseudo-SQL statements if the --verbose option is also
given; 'auto' prints base64 only when necessary (i.e.,
for row-based events and format description events). If
no --base64-output[=name] option is given at all, the
default is 'auto'.
--bind-address=name IP address to bind to.
--character-sets-dir=name
Directory for character set files.
-d, --database=name List entries for just this database (local log only).
--rewrite-db=name Rewrite the row event to point so that it can be applied
to a new database
-#, --debug[=#] This is a non-debug version. Catch this and exit.
--debug-check This is a non-debug version. Catch this and exit.
--debug-info This is a non-debug version. Catch this and exit.
--default-auth=name Default authentication client-side plugin to use.
-D, --disable-log-bin
Disable binary log. This is useful, if you enabled
--to-last-log and are sending the output to the same
MySQL server. This way you could avoid an endless loop.
You would also like to use it when restoring after a
crash to avoid duplication of the statements you already
have. NOTE: you will need a SUPER privilege to use this
option.
-F, --force-if-open Force if binlog was not closed properly.
(Defaults to on; use --skip-force-if-open to disable.)
-f, --force-read Force reading unknown binlog events.
-H, --hexdump Augment output with hexadecimal and ASCII event dump.
-h, --host=name Get the binlog from server.
-i, --idempotent Notify the server to use idempotent mode before applying
Row Events
-l, --local-load=name
Prepare local temporary files for LOAD DATA INFILE in the
specified directory.
-o, --offset=# Skip the first N entries.
-p, --password[=name]
Password to connect to remote server.
--plugin-dir=name Directory for client-side plugins.
-P, --port=# Port number to use for connection or 0 for default to, in
order of preference, my.cnf, $MYSQL_TCP_PORT,
/etc/services, built-in default (3306).
--protocol=name The protocol to use for connection (tcp, socket, pipe,
memory).
-R, --read-from-remote-server
Read binary logs from a MySQL server. This is an alias
for read-from-remote-master=BINLOG-DUMP-NON-GTIDS.
--read-from-remote-master=name
Read binary logs from a MySQL server through the
COM_BINLOG_DUMP or COM_BINLOG_DUMP_GTID commands by
setting the option to either BINLOG-DUMP-NON-GTIDS or
BINLOG-DUMP-GTIDS, respectively. If
--read-from-remote-master=BINLOG-DUMP-GTIDS is combined
with --exclude-gtids, transactions can be filtered out on
the master avoiding unnecessary network traffic.
--raw Requires -R. Output raw binlog data instead of SQL
statements, output is to log files.
-r, --result-file=name
Direct output to a given file. With --raw this is a
prefix for the file names.
--secure-auth Refuse client connecting to server if it uses old
(pre-4.1.1) protocol. Deprecated. Always TRUE
--server-id=# Extract only binlog entries created by the server having
the given id.
--server-id-bits=# Set number of significant bits in server-id
--set-charset=name Add 'SET NAMES character_set' to the output.
-s, --short-form Just show regular queries: no extra info and no row-based
events. This is for testing only, and should not be used
in production systems. If you want to suppress
base64-output, consider using --base64-output=never
instead.
-S, --socket=name The socket file to use for connection.
--ssl-mode=name SSL connection mode.
--ssl Deprecated. Use --ssl-mode instead.
(Defaults to on; use --skip-ssl to disable.)
--ssl-verify-server-cert
Deprecated. Use --ssl-mode=VERIFY_IDENTITY instead.
--ssl-ca=name CA file in PEM format.
--ssl-capath=name CA directory.
--ssl-cert=name X509 cert in PEM format.
--ssl-cipher=name SSL cipher to use.
--ssl-key=name X509 key in PEM format.
--ssl-crl=name Certificate revocation list.
--ssl-crlpath=name Certificate revocation list path.
--tls-version=name TLS version to use, permitted values are: TLSv1, TLSv1.1,
TLSv1.2
--server-public-key-path=name
File path to the server public RSA key in PEM format.
--get-server-public-key
Get server public key
--start-datetime=name
Start reading the binlog at first event having a datetime
equal or posterior to the argument; the argument must be
a date and time in the local time zone, in any format
accepted by the MySQL server for DATETIME and TIMESTAMP
types, for example: 2004-12-25 11:25:56 (you should
probably use quotes for your shell to set it properly).
-j, --start-position=#
Start reading the binlog at position N. Applies to the
first binlog passed on the command line.
--stop-datetime=name
Stop reading the binlog at first event having a datetime
equal or posterior to the argument; the argument must be
a date and time in the local time zone, in any format
accepted by the MySQL server for DATETIME and TIMESTAMP
types, for example: 2004-12-25 11:25:56 (you should
probably use quotes for your shell to set it properly).
--stop-never Wait for more data from the server instead of stopping at
the end of the last log. Implicitly sets --to-last-log
but instead of stopping at the end of the last log it
continues to wait till the server disconnects.
--stop-never-slave-server-id=#
The slave server_id used for --read-from-remote-server
--stop-never. This option cannot be used together with
connection-server-id.
--connection-server-id=#
The slave server_id used for --read-from-remote-server.
This option cannot be used together with
stop-never-slave-server-id.
--stop-position=# Stop reading the binlog at position N. Applies to the
last binlog passed on the command line.
-t, --to-last-log Requires -R. Will not stop at the end of the requested
binlog but rather continue printing until the end of the
last binlog of the MySQL server. If you send the output
to the same MySQL server, that may lead to an endless
loop.
-u, --user=name Connect to the remote server as username.
-v, --verbose Reconstruct pseudo-SQL statements out of row events. -v
-v adds comments on column data types.
-V, --version Print version and exit.
--open-files-limit=#
Used to reserve file descriptors for use by this program.
-c, --verify-binlog-checksum
Verify checksum binlog events.
--binlog-row-event-max-size=#
The maximum size of a row-based binary log event in
bytes. Rows will be grouped into events smaller than this
size if possible. This value must be a multiple of 256.
--skip-gtids Do not preserve Global Transaction Identifiers; instead
make the server execute the transactions as if they were
new.
--include-gtids=name
Print events whose Global Transaction Identifiers were
provided.
--exclude-gtids=name
Print all events but those whose Global Transaction
Identifiers were provided.
Variables (--variable-name=value)
and boolean options {FALSE|TRUE} Value (after reading options)
- --no-defaults: 默認charset問題
示例
# 普通權限的用戶只讀，不能寫數據
set global read_only=1;
# 查看操作記錄信息
mysql> show binlog events in 'mysql-bin.000002';
# 恢復指定位置的操作
mysqlbinlog --start-position=120 --stop-position=520 --database=demo /var/lib/mysql/mysql-bin.000003 | /usr/bin/mysql -u root -p -v demo
# 基於時間點恢復
/usr/bin/mysqlbinlog --start-datetime="2021-06-27 20:57:55" --stop-datetime="2021-06-27 20:58:18" --database=demo /var/lib/mysql/mysql-bin.000009 | /usr/bin/mysql -uroot -p -v demo
MyFlash
美團點評的開源MySQL閃回工具。
安裝
推薦下載源碼之后，進行動態編譯鏈接安裝
動態編譯鏈接
cc -w `pkg-config --cflags --libs glib-2.0` source/binlogParseGlib.c -o binary/flashback
靜態編譯鏈接
gcc -w -g `pkg-config --cflags glib-2.0` source/binlogParseGlib.c -o binary/flashback /usr/lib64/libglib-2.0.a -lrt
注意：
- 確認目標設備上 glib庫的版本和位置(ldd --version)
使用
語法
cd binary
./flashback --help
Usage:
flashback [OPTION...]
Help Options:
-?, --help Show help options
Application Options:
--databaseNames databaseName to apply. if multiple, seperate by comma(,)
--tableNames tableName to apply. if multiple, seperate by comma(,)
--start-position start position
--stop-position stop position
--start-datetime start time (format %Y-%m-%d %H:%M:%S)
--stop-datetime stop time (format %Y-%m-%d %H:%M:%S)
--sqlTypes sql type to filter . support INSERT, UPDATE ,DELETE. if multiple, seperate by comma(,)
--maxSplitSize max file size after split, the uint is M
--binlogFileNames binlog files to process. if multiple, seperate by comma(,)
--outBinlogFileNameBase output binlog file name base
--logLevel log level, available option is debug,warning,error
--include-gtids gtids to process
--exclude-gtids gtids to skip
參數說明
-
databaseNames
指定需要回滾的數據庫名。多個數據庫可以用“,”隔開。如果不指定該參數，相當於指定了所有數據庫。
-
tableNames
指定需要回滾的表名。多個表可以用“,”隔開。如果不指定該參數，相當於指定了所有表。
-
start-position
指定回滾開始的位置。如不指定，從文件的開始處回滾。請指定正確的有效的位置，否則無法回滾
-
stop-position
指定回滾結束的位置。如不指定，回滾到文件結尾。請指定正確的有效的位置，否則無法回滾
-
start-datetime
指定回滾的開始時間。注意格式必須是 %Y-%m-%d %H:%M:%S。 如不指定，則不限定時間
-
stop-datetime
指定回滾的結束時間。注意格式必須是 %Y-%m-%d %H:%M:%S。 如不指定，則不限定時間
-
sqlTypes
指定需要回滾的sql類型。目前支持的過濾類型是INSERT, UPDATE ,DELETE。多個類型可以用“,”隔開。
-
maxSplitSize
一旦指定該參數，對文件進行固定尺寸的分割（單位為M），過濾條件有效，但不進行回滾操作。該參數主要用來將大的binlog文件切割，防止單次應用的binlog尺寸過大，對線上造成壓力
-
binlogFileNames
指定需要回滾的binlog文件，目前只支持單個文件，后續會增加多個文件支持
-
outBinlogFileNameBase
指定輸出的binlog文件前綴，如不指定，則默認為binlog_output_base.flashback
-
logLevel
僅供開發者使用，默認級別為error級別。在生產環境中不要修改這個級別，否則輸出過多
-
include-gtids
指定需要回滾的gtid,支持gtid的單個和范圍兩種形式。
-
exclude-gtids
指定不需要回滾的gtid，用法同include-gtids
示例
回滾整個文件
# 閃回結果存放到binlog_output_base.flashback中
./flashback --binlogFileNames=haha.000041
# 應用閃回的日志
mysqlbinlog binlog_output_base.flashback | mysql -h<host> -u<user> -p
回滾該文件中的所有insert語句
./flashback --sqlTypes='INSERT' --binlogFileNames=haha.000041
mysqlbinlog binlog_output_base.flashback | mysql -h<host> -u<user> -p
回滾大文件
#回滾
./flashback --binlogFileNames=haha.000042
#切割大文件
./flashback --maxSplitSize=1 --binlogFileNames=binlog_output_base.flashback
#應用
mysqlbinlog binlog_output_base.flashback.000001 | mysql -h<host> -u<user> -p
...
mysqlbinlog binlog_output_base.flashback.<N> | mysql -h<host> -u<user> -p
綜合測試用例
測試表結構
CREATE TABLE `testFlashback2` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`nameShort` varchar(20) DEFAULT NULL,
`nameLong` varchar(260) DEFAULT NULL,
`amount` decimal(19,9) DEFAULT NULL,
`amountFloat` float DEFAULT NULL,
`amountDouble` double DEFAULT NULL,
`createDatetime6` datetime(6) DEFAULT NULL,
`createDatetime` datetime DEFAULT NULL,
`createTimestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`nameText` text,
`nameBlob` blob,
`nameMedium` mediumtext,
PRIMARY KEY (`id`)
) ENGINE=InnoDB
插入&回滾
插入數據
flush logs
insert into testFlashback2(nameShort,nameLong,amount,amountFloat,amountDouble,createDatetime6,createDatetime,createTimestamp,nameText,nameBlob,nameMedium) values('aaa','bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',10.5,10.6,10.7,'2017-10-26 10:00:00','2017-10-26 10:00:00','2017-10-26 10:00:00','cccc','dddd','eee');
insert into testFlashback2(nameShort,nameLong,amount,amountFloat,amountDouble,createDatetime6,createDatetime,createTimestamp,nameText,nameBlob,nameMedium) values('aaa','bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',10.5,10.6,10.7,'2017-10-26 10:00:00','2017-10-26 10:00:00','2017-10-26 10:00:00','cccc','dddd','eee');
flush logs;
回滾數據
./binary/flashback --binlogFileNames=/var/lib/mysql/haha.000048
# 在當前運行目錄下產生binlog_output_base.flashback文件
mysqlbinlog --skip-gtids binlog_output_base.flashback | mysql --socket=/var/lib/mysql/mysql.sock test
刪除&回滾
刪除數據
delete from testFlashback2;
insert into testFlashback2(nameShort,nameLong,amount,amountFloat,amountDouble,createDatetime6,createDatetime,createTimestamp,nameText,nameBlob,nameMedium) values('aaa','bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',10.5,10.6,10.7,'2017-10-26 10:00:00','2017-10-26 10:00:00','2017-10-26 10:00:00','cccc','dddd','eee');
flush logs;
delete from testFlashback2;
回滾數據
./binary/flashback --binlogFileNames=/var/lib/mysql/haha.000050
# 在當前運行目錄下產生binlog_output_base.flashback文件
mysqlbinlog --skip-gtids binlog_output_base.flashback | mysql --socket=/var/lib/mysql/mysql.sock test
更新&回滾
更新數據
delete from testFlashback2;
insert into testFlashback2(nameShort,nameLong,amount,amountFloat,amountDouble,createDatetime6,createDatetime,createTimestamp,nameText,nameBlob,nameMedium) values('aaa','bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',10.111,10.6,10.7,'2017-10-26 10:00:00','2017-10-26 10:00:00','2017-10-26 10:00:00','cccc','dddd','eee');
flush logs;
mysql> checksum table testFlashback2;
+---------------------+-----------+
| Table | Checksum |
+---------------------+-----------+
| test.testFlashback2 | 717087411 |
+---------------------+-----------+
update testFlashback2 set amount=10.222;
mysql> checksum table testFlashback2;
+---------------------+------------+
| Table | Checksum |
+---------------------+------------+
| test.testFlashback2 | 3797190846 |
+---------------------+------------+
回滾數據
/binary/flashback --binlogFileNames=/var/lib/mysql/haha.000052
mysqlbinlog --skip-gtids binlog_output_base.flashback | mysql --socket=/var/lib/mysql/mysql.sock test
binlog2sql
binlog2sql是一個開源的Python開發的MySQL Binlog解析工具。根據選項不同，可以得到原始SQL、回滾SQL、去除主鍵的INSERT SQL等。
要求
MySQL server必須設置以下參數
[mysqld]
server_id = 1
log_bin = /var/log/mysql/mysql-bin.log
max_binlog_size = 1G
binlog_format = row
binlog_row_image = full
- 參數 binlog_row_image 必須為FULL，暫不支持MINIMAL
用戶所需最小權限
select, super/replication client, replication slave
-- 建議授權
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO USER_NAME;
說明
- select：需要讀取server端information_schema.COLUMNS表，獲取表結構的元信息，拼接成可視化的sql語句
- super/replication client：兩個權限都可以，需要執行'SHOW MASTER STATUS', 獲取server端的binlog列表
- replication slave：通過BINLOG_DUMP協議獲取binlog內容的權限
mysql server必須開啟，不支持離線解析
通過 BINLOG_DUMP 協議來獲取 binlog 內容，需要讀取server端 information_schema.COLUMNS 表，來獲取表結構的元信息，才能拼接成 SQL 語句。因此，必須開啟mysql server。
安裝配置
依賴
- Python 2.7, 3.4+
- MySQL 5.6, 5.7
安裝
git clone https://github.com/danfengcao/binlog2sql.git && cd binlog2sql
pip install -r requirements.txt
使用
用法
解析出標准SQL
python binlog2sql.py -h127.0.0.1 -P3306 -uadmin -p'admin' -dtest -t test3 test4 --start-file='mysql-bin.000002'
解析回滾SQl
python binlog2sql.py --flashback -h127.0.0.1 -P3306 -uadmin -p'admin' -dtest -ttest3 --start-file='mysql-bin.000002' --start-position=763 --stop-position=1147
選項說明
解析模式
-
--stop-never ：持續解析binlog。可選。默認False，同步至執行命令時最新的binlog位置。
-
-K, --no-primary-key ：對INSERT語句去除主鍵。可選。默認False
-
-B, --flashback ：生成回滾SQL，可解析大文件，不受內存限制。可選。默認False。與stop-never或no-primary-key不能同時添加。
-
--back-interval ：在-B模式下，每打印一千行回滾SQL，加一句SLEEP多少秒，如不想加SLEEP，請設為0。可選。默認1.0。
解析范圍
- --start-file： 起始解析文件，只需文件名，無需全路徑 。必須。
- --start-position/--start-pos： 起始解析位置。可選。默認為start-file的起始位置。
- --stop-file/--end-file ：終止解析文件。可選。默認為start-file同一個文件。若解析模式為stop-never，此選項失效。
- --stop-position/--end-pos ：終止解析位置。可選。默認為stop-file的最末位置；若解析模式為stop-never，此選項失效。
- --start-datetime ：起始解析時間，格式'%Y-%m-%d %H:%M:%S'。可選。默認不過濾。
- --stop-datetime ：終止解析時間，格式'%Y-%m-%d %H:%M:%S'。可選。默認不過濾。
過濾對象
-
-d, --databases ：只解析目標db的sql，多個庫用空格隔開，如-d db1 db2。可選。默認為空。
-
-t, --tables ：只解析目標table的sql，多張表用空格隔開，如-t tbl1 tbl2。可選。默認為空。
-
--only-dml ：只解析dml，忽略ddl。可選。默認False。
-
--sql-type ：只解析指定類型，支持INSERT, UPDATE, DELETE。多個類型用空格隔開，如--sql-type INSERT DELETE。可選。默認為增刪改都解析。用了此參數但沒填任何類型，則三者都不解析。
示例
創建測試數據
--構造測試表
create table tbl
(
id int primary key,
name varchar(30) not null,
birthday date not null
);
-- 插入3條數據
insert into tbl values(1,'小明','1993-01-02');
insert into tbl values(2,'小華','1994-08-15');
insert into tbl values(3,'小麗','1995-07-12');
-- 模擬誤刪除數據
delete from tbl;
恢復數據
查看binlog日志
show master status;
解析出標准SQL
python binlog2sql/binlog2sql.py -h127.0.0.1 -P3306 -uadmin -p'admin' -dtest -ttbl --start-file='mysql-bin.000052' --start-datetime='2016-12-13 20:25:00' --stop-datetime='2016-12-13 20:30:00'
使用flashback模式生成回滾sql
python binlog2sql/binlog2sql.py -h127.0.0.1 -P3306 -uadmin -p'admin' -dtest -ttbl --start-file='mysql-bin.000052' --start-position=3346 --stop-position=3556 -B > rollback.sql | cat
應用回滾數據
mysql -h127.0.0.1 -P3306 -uadmin -p'admin' < rollback.sql
my2sql
go版MySQL binlog解析工具，通過解析MySQL binlog ，可以生成原始SQL、回滾SQL、去除主鍵的INSERT SQL等，也可以生成DML統計信息。它基於my2fback、binlog_rollback工具二次開發而來。
限制
- 使用回滾/閃回功能時，binlog格式必須為row,且binlog_row_image=full， DML統計以及大事務分析不受影響
- 只能回滾DML， 不能回滾DDL
- 支持指定-tl時區來解釋binlog中time/datetime字段的內容。開始時間-start-datetime與結束時間-stop-datetime也會使用此指定的時區， 但注意此開始與結束時間針對的是binlog event header中保存的unix timestamp。結果中的額外的datetime時間信息都是binlog event header中的unix timestamp
- 此工具是偽裝成從庫拉取binlog，需要連接數據庫的用戶有SELECT, REPLICATION SLAVE, REPLICATION CLIENT權限
- MySQL8.0版本需要在配置文件中加入default_authentication_plugin =mysql_native_password，用戶密碼認證必須是mysql_native_password才能解析
安裝配置
編譯
git clone https://github.com/liuhr/my2sql.git
cd my2sql/
go build .
已編譯二進制文件
https://github.com/liuhr/my2sql/blob/master/releases/centOS_release_7.x/my2sql
使用
語法
參數說明
-
-U: 優先使用unique key作為where條件，默認false
-
-mode:
- repl: 偽裝成從庫解析binlog文件, 默認repl
- file: 離線解析binlog文件
-
-local-binlog-file: 當指定-mode=file 參數時，需要指定-local-binlog-file binlog文件相對路徑或絕對路徑,可以連續解析多個binlog文件，只需要指定起始文件名，程序會自動持續解析下個文件
-
-add-extraInfo: 是否把database/table/datetime/binlogposition...信息以注釋的方式加入生成的每條sql前，默認false
-
-big-trx-row-limit n: 找出滿足n條sql的事務，默認500條
-
-databases 、 -tables: 庫及表條件過濾, 以逗號分隔
-
-sql: 要解析的sql類型，可選參數insert、update、delete，默認全部解析
-
-doNotAddPrifixDb: 默認生成insert into db1.tb1 (x1, x1) values (y1, y1)類sql，也可以生成不帶庫名的sql
-
-file-per-table: 為每個表生成一個sql文件
-
-full-columns: 生成的sql是否帶全列信息，默認false
-
-ignorePrimaryKeyForInsert: 生成的insert語句是否去掉主鍵，默認false
-
-output-dir: 將生成的結果存放到制定目錄
-
-output-toScreen: 將生成的結果打印到屏幕，默認寫到文件
-
-threads: 線程數，默認8個
-
-work-type:
- 2sql：生成原始sql
- rollback：生成回滾sql
- stats：只統計DML、事務信息
解析出標准SQL
根據時間點解析出標准SQL
#偽裝成從庫解析binlog
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode repl -work-type 2sql -start-file mysql-bin.011259 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -output-dir ./tmpdir
#直接讀取binlog文件解析
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode file -local-binlog-file ./mysql-bin.011259 -work-type 2sql -start-file mysql-bin.011259 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -output-dir ./tmpdir
根據pos點解析出標准SQL
#偽裝成從庫解析binlog
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode repl -work-type 2sql -start-file mysql-bin.011259 -start-pos 4 -stop-file mysql-bin.011259 -stop-pos 583918266 -output-dir ./tmpdir
#直接讀取binlog文件解析
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode file -local-binlog-file ./mysql-bin.011259 -work-type 2sql -start-file mysql-bin.011259 -start-pos 4 -stop-file mysql-bin.011259 -stop-pos 583918266 -output-dir ./tmpdir
解析出回滾SQL
根據時間點解析出回滾SQL
#偽裝成從庫解析binlog
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode repl -work-type rollback -start-file mysql-bin.011259 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -output-dir ./tmpdir
#直接讀取binlog文件解析
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode file -local-binlog-file ./mysql-bin.011259 -work-type rollback -start-file mysql-bin.011259 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -output-dir ./tmpdir
根據pos點解析出回滾SQL
#偽裝成從庫解析binlog
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode repl -work-type rollback -start-file mysql-bin.011259 -start-pos 4 -stop-file mysql-bin.011259 -stop-pos 583918266 -output-dir ./tmpdir
#直接讀取binlog文件解析
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode file -local-binlog-file ./mysql-bin.011259 -work-type rollback -start-file mysql-bin.011259 -start-pos 4 -stop-file mysql-bin.011259 -stop-pos 583918266 -output-dir ./tmpdir
統計DML以及大事務
統計時間范圍各個表的DML操作數量，統計一個事務大於500條、時間大於300秒的事務
#偽裝成從庫解析binlog
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode repl -work-type stats -start-file mysql-bin.011259 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -big-trx-row-limit 500 -long-trx-seconds 300 -output-dir ./tmpdir
#直接讀取binlog文件解析
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode file -local-binlog-file ./mysql-bin.011259 -work-type stats -start-file mysql-bin.011259 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -big-trx-row-limit 500 -long-trx-seconds 300 -output-dir ./tmpdir
統計一段pos點范圍各個表的DML操作數量，統計一個事務大於500條、時間大於300秒的事務
#偽裝成從庫解析binlog
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode repl -work-type stats -start-file mysql-bin.011259 -start-pos 4 -stop-file mysql-bin.011259 -stop-pos 583918266 -big-trx-row-limit 500 -long-trx-seconds 300 -output-dir ./tmpdir
#直接讀取binlog文件解析
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode file -local-binlog-file ./mysql-bin.011259 -work-type stats -start-file mysql-bin.011259 -start-pos 4 -stop-file mysql-bin.011259 -stop-pos 583918266 -big-trx-row-limit 500 -long-trx-seconds 300 -output-dir ./tmpdir
從某一個pos點解析出標准SQL，並且持續打印到屏幕
#偽裝成從庫解析binlog
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode repl -work-type 2sql -start-file mysql-bin.011259 -start-pos 4 -output-toScreen
示例
誤刪整張表數據，需要緊急回滾
測試數據
CREATE TABLE `tb1` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) DEFAULT NULL,
--`add_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP COMMENT '添加的時間',
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;
insert into tb1 values(1, 'biu');
insert into tb1 values(2, 'biao');
commit;
-- 查看測試表校驗值
checksum table tb1;
flush logs;
-- 查看當前binlog
show master status;
-- 刪除表數據
delete from tb1;
commit;
生成標准SQL
#偽裝成從庫解析binlog
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -databases devdb -tables tb1 -mode repl -work-type 2sql -start-file mysql-bin.000046 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -output-dir /tmp
#直接讀取binlog文件解析
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -databases devdb -tables tb1 -mode file -local-binlog-file ./mysql-bin.000046 -work-type 2sql -start-file mysql-bin.000046 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -output-dir /tmp
解析出回滾SQL
#偽裝成從庫解析binlog
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -databases devdb -tables tb1 -mode repl -work-type rollback -start-file mysql-bin.000046 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -output-dir /tmp
#直接讀取binlog文件解析
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -databases devdb -tables tb1 -mode file -local-binlog-file ./mysql-bin.000046 -work-type rollback -start-file mysql-bin.000046 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -output-dir /tmp
-
查看回滾SQL
cat rollback.46.sql
應用回滾SQL恢復數據
mysql -uroot -proot -P3306 -h127.0.0.1 devdb < /tmp/rollback.46.sql
Maxwell
Maxwell 是一個讀取 MySQL binlogs 並將修改行字段的更新寫入 Kafka, Kinesis, RabbitMQ, Google Cloud Pub/Sub 或 Redis (Pub/Sub or LPUSH) 以作為 JSON 的應用程序。
my2fback
my2fback 實現了基於row格式binlog的回滾閃回功能，讓誤刪除或者誤更新數據，可以不停機不使用備份而快速回滾誤操作。也可以解釋binlog（支持非row格式binlog）生成易讀的SQL。
限制
- 使用回滾/閃回功能時，binlog格式必須為row,且binlog_row_image=full， 其它功能支持非row格式binlog
- 只能回滾DML， 不能回滾DDL
- 支持V4格式的binlog， V3格式的沒測試過，測試與使用結果顯示，mysql5.1，mysql5.5, mysql5.6與mysql5.7的binlog均支持
- 支持指定-tl時區來解釋binlog中time/datetime字段的內容。開始時間-sdt與結束時間-edt也會使用此指定的時區，
- 但注意此開始與結束時間針對的是binlog event header中保存的unix timestamp。結果中的額外的datetime時間信息都是binlog event header中的unix timestamp
- decimal字段使用float64來表示， 但不損失精度
- 所有字符類型字段內容按golang的utf8(相當於mysql的utf8mb4)來表示
安裝配置
使用GO>=1.11.x版本來編譯
開啟GO111MODULE參數
編譯linux 平台
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o releases/my2fback -ldflags "-s -w" main.go
編譯windows 平台
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o releases/my2fback -ldflags "-s -w" main.go
沒有開啟GO111MODULE參數
編譯linux 平台
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o releases/my2fback -ldflags "-s -w" main.go
編譯windows 平台
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o releases/my2fback -ldflags "-s -w" main.go
使用
語法
my2fback -h
my2fback V2.0 By WangJiemin.
E_mail: 278667010@qq.com
*****************************************************************************************************
* system_command: /usr/local/bin/my2fback *
* system_goos: linux *
* system_arch: amd64 *
* hostname: test_dbs2.yz.babytree-ops.org *
* hostaddress: 10.10.1.221 *
* blog: https://jiemin.wang *
* read binlog from master, work as a fake slave: ./my2fback -m repl opts... *
* read binlog from local filesystem: ./my2fback -m file opts... mysql-bin.000010 *
*****************************************************************************************************
-C works with -w='stats', keep analyzing transations to last binlog for -m=file, and keep analyzing for -m=repl
-H string
master host, DONOT need to specify when -w=stats. if mode is file, it can be slave or other mysql contains same schema and table structure, not only master. default 127.0.0.1 (default "127.0.0.1")
-I for insert statement when -wtype=2sql, ignore primary key
-M string
valid options are: mysql,mariadb. server of binlog, mysql or mariadb, default mysql (default "mysql")
-P uint
master port, default 3306. DONOT need to specify when -w=stats (default 3306)
-S string
mysql socket file
-U prefer to use unique key instead of primary key to build where condition for delete/update sql
-a Works with -w=2sql|rollback. for update sql, include unchanged columns. for update and delete, use all columns to build where condition.
default false, this is, use changed columns to build set part, use primary/unique key to build where condition
-b int
transaction with affected rows greater or equal to this value is considerated as big transaction. Valid values range from 10 to 30000, default 500 (default 500)
-d Works with -w=2sql|rollback. Prefix table name with database name in sql, ex: insert into db1.tb1 (x1, x1) values (y1, y1). Default true (default true)
-dbs string
only parse database which match any of these regular expressions. The regular expression should be in lower case because database name is translated into lower case and then matched against it.
Multi regular expressions is seperated by comma, default parse all databases. Useless when -w=stats
-dj string
dump table structure to this file. default tabSchame.json (default "tabSchame.json")
-e Works with -w=2sql|rollback. Print database/table/datetime/binlogposition...info on the line before sql, default false
-ebin string
binlog file to stop reading
-edt string
Stop reading the binlog at first event having a datetime equal or posterior to the argument, it should be like this: "2004-12-25 11:25:56"
-epos uint
Stop reading the binlog at position
-f Works with -w=2sql|rollback. one file for one table if true, else one file for all tables. default false. Attention, always one file for one binlog
-i int
works with -w='stats', print stats info each PrintInterval. Valid values range from 1 to 600, default 30 (default 30)
-ies string
for sql which is error to parsed and matched by this regular expression, just print error info, skip it and continue parsing, otherwise stop parsing and exit.
The regular expression should be in lower case, because sql is translated into lower case and then matched against it. (default "^create definer.+trigger")
-k Works with -w=2sql|rollback. wrap result statements with 'begin...commit|rollback'
-l int
transaction with duration greater or equal to this value is considerated as long transaction. Valid values range from 1 to 3600, default 300 (default 300)
-m string
valid options are: repl,file. repl: as a slave to get binlogs from master. file: get binlogs from local filesystem. default file (default "file")
-mid uint
works with -m=repl, this program replicates from master as slave to read binlogs. Must set this server id unique from other slaves, default 1113306 (default 1113306)
-o string
result output dir, default current work dir. Attension, result files could be large, set it to a dir with large free space
-oj
Only use table structure from -rj, do not get or merge table struct from mysql
-ors
for mysql>=5.6.2 and binlog_rows_query_log_events=on, if set, output original sql. default false
-p string
mysql user password. DONOT need to specify when -w=stats
-r int
Works with -w=2sql|rollback. rows for each insert sql. Valid values range from 1 to 500, default 30 (default 30)
-rj string
Works with -w=2sql|rollback, read table structure from this file and merge from mysql
-sbin string
binlog file to start reading
-sdt string
Start reading the binlog at first event having a datetime equal or posterior to the argument, it should be like this: "2004-12-25 11:25:56"
-spos uint
start reading the binlog at position
-sql string
valid options are: insert,update,delete. only parse these types of sql, comma seperated, valid types are: insert, update, delete; default is all(insert,update,delete)
-stsql
when -w=2sql, also parse plain sql and write into result file even if binlog_format is not row. default false
-t uint
Works with -w=2sql|rollback. threads to run, default 4 (default 2)
-tbs string
only parse table which match any of these regular expressions.The regular expression should be in lower case because database name is translated into lower case and then matched against it.
Multi regular expressions is seperated by comma, default parse all tables. Useless when -w=stats
-tl string
time location to parse timestamp/datetime column in binlog, such as Asia/Shanghai. default Local (default "Local")
-u string
mysql user. DONOT need to specify when -w=stats
-v print version
-w string
valid options are: tbldef,stats,2sql,rollback. tbldef: only get table definition structure; 2sql: convert binlog to sqls, rollback: generate rollback sqls, stats: analyze transactions. default: stats (default "stats")
常用參數
-m string
valid options are: repl,file. repl: as a slave to get binlogs from master. file: get binlogs from local filesystem. default file (default "file")
relp: 模仿 SLAVE 的IO_THREAD連接到MASTER獲取BINLOG EVENT
file: 解析本地的BINLOG(default: file)
-w string
valid options are: tbldef,stats,2sql,rollback. tbldef: only get table definition structure; 2sql: convert binlog to sqls, rollback: generate rollback sqls, stats: analyze transactions. default: stats (default "stats")
2sql: 解析成SQL語句
rollback: 解析為回滾語句
-M string
valid options are: mysql,mariadb. server of binlog, mysql or mariadb, default mysql (default "mysql")
選擇是MySQL還是Mariadb, 不選擇默認為MySQL
-e Works with -w=2sql|rollback. Print database/table/datetime/binlogposition...info on the line before sql, default false
在sql之前的行上打印database/table/datetime/binlogposition...info，默認為false
-f Works with -w=2sql|rollback. one file for one table if true, else one file for all tables. default false. Attention, always one file for one binlog
如果為true，則為一個表的一個文件，否則為所有表的一個文件。默認為false。注意，一個binlog總是一個文件
-r int
Works with -w=2sql|rollback. rows for each insert sql. Valid values range from 1 to 500, default 30 (default 30)
INSERT SQL 語句每一行包含的values的行數
-t uint
Works with -w=2sql|rollback. threads to run, default 4 (default 2)
開啟幾個thread進行來執行解析2sql|rollback
-o string
result output dir, default current work dir. Attension, result files could be large, set it to a dir with large free space
輸入的目錄
-k Works with -w=2sql|rollback. wrap result statements with 'begin...commit|rollback'
使用-w = 2sql | rollback。使用'begin ... commit | rollback'包裝結果語句
-l int
transaction with duration greater or equal to this value is considerated as long transaction. Valid values range from 1 to 3600, default 300 (default 300)
-b int
transaction with affected rows greater or equal to this value is considerated as big transaction. Valid values range from 10 to 30000, default 500 (default 500)
-dWorks with -w=2sql|rollback. Prefix table name with database name in sql, ex: insert into db1.tb1 (x1, x1) values (y1, y1). Default true (default true)
使用-w=2sql|rollback。在sql中具有數據庫名稱的前綴表名
示例
file本地方式解析binlog
./my2fback -m file -w 2sql -M mysql -t 6 -H ***.***.***.*** -u test -p test -dbs babytree -tbs userbaby -e -f -d -r 20 -k -b 100 -l 10 -o /data/bak/20190626/tosql /data/bak/20190626/mysql-bin.002938
binlog_rollback
回滾/閃回， 前滾， DML分析報告， DDL信息
binlog_inspector
回滾/閃回，前滾， 分析各表DML情況， 找出長事務與大事務