pt-online-schema-change用於MySQL的在線DDL。
下面結合官方文檔和general log來分析其實現原理。
測試表
mysql> show create table t2\G *************************** 1. row *************************** Table: t2 Create Table: CREATE TABLE `t2` ( `id` int(11) NOT NULL AUTO_INCREMENT, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=1005764 DEFAULT CHARSET=utf8 1 row in set (0.19 sec)
該表中只有1列，id，自增主鍵。
其中，表中已經存在一部分數據
mysql> select count(*) from t2; +----------+ | count(*) | +----------+ | 1005763 | +----------+ 1 row in set (0.31 sec)
利用pt-online-schema-change對該表新增一列
# pt-online-schema-change --execute --alter "ADD COLUMN c1 DATETIME" D=test,t=t2
Found 2 slaves: test hbase Will check slave lag on: test hbase Operation, tries, wait: analyze_table, 10, 1 copy_rows, 10, 0.25 create_triggers, 10, 1 drop_triggers, 10, 1 swap_tables, 10, 1 update_foreign_keys, 10, 1 Altering `test`.`t2`... Creating new table... CREATE TABLE `test`.`_t2_new` ( `id` int(11) NOT NULL AUTO_INCREMENT, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=1005764 DEFAULT CHARSET=utf8 Created new table test._t2_new OK. Waiting forever for new table `test`.`_t2_new` to replicate to test... Altering new table... ALTER TABLE `test`.`_t2_new` ADD COLUMN c1 DATETIME Altered `test`.`_t2_new` OK. 2016-11-21T12:49:18 Creating triggers... CREATE TRIGGER `pt_osc_test_t2_del` AFTER DELETE ON `test`.`t2` FOR EACH ROW DELETE IGNORE FROM `test`.`_t2_new` WHERE `test`.`_t2_ne w`.`id` <=> OLD.`id`CREATE TRIGGER `pt_osc_test_t2_upd` AFTER UPDATE ON `test`.`t2` FOR EACH ROW REPLACE INTO `test`.`_t2_new` (`id`) VALUES (NEW.`id`) CREATE TRIGGER `pt_osc_test_t2_ins` AFTER INSERT ON `test`.`t2` FOR EACH ROW REPLACE INTO `test`.`_t2_new` (`id`) VALUES (NEW.`id`) 2016-11-21T12:49:18 Created triggers OK. 2016-11-21T12:49:18 Copying approximately 1005075 rows... INSERT LOW_PRIORITY IGNORE INTO `test`.`_t2_new` (`id`) SELECT `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= ?)) AND ( (`id` <= ?)) LOCK IN SHARE MODE /*pt-online-schema-change 2352 copy nibble*/SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= ?)) ORDER BY `id` LIMIT ?, 2 /*next chun k boundary*/Copying `test`.`t2`: 40% 00:44 remain Copying `test`.`t2`: 82% 00:12 remain 2016-11-21T12:50:31 Copied rows OK. 2016-11-21T12:50:31 Analyzing new table... 2016-11-21T12:50:32 Swapping tables... RENAME TABLE `test`.`t2` TO `test`.`_t2_old`, `test`.`_t2_new` TO `test`.`t2` 2016-11-21T12:50:35 Swapped original and new tables OK. 2016-11-21T12:50:35 Dropping old table... DROP TABLE IF EXISTS `test`.`_t2_old` 2016-11-21T12:50:36 Dropped old table `test`.`_t2_old` OK. 2016-11-21T12:50:36 Dropping triggers... DROP TRIGGER IF EXISTS `test`.`pt_osc_test_t2_del`; DROP TRIGGER IF EXISTS `test`.`pt_osc_test_t2_upd`; DROP TRIGGER IF EXISTS `test`.`pt_osc_test_t2_ins`; 2016-11-21T12:50:36 Dropped triggers OK. Successfully altered `test`.`t2`.
查看general log中的輸出
161017 11:22:56 1052 Connect root@localhost on test 1052 Query set autocommit=1 1052 Query SHOW VARIABLES LIKE 'innodb\_lock_wait_timeout' 1052 Query SET SESSION innodb_lock_wait_timeout=1 1052 Query SHOW VARIABLES LIKE 'lock\_wait_timeout' 1052 Query SET SESSION lock_wait_timeout=60 1052 Query SHOW VARIABLES LIKE 'wait\_timeout' 1052 Query SET SESSION wait_timeout=10000 1052 Query SELECT @@SQL_MODE 1052 Query SET @@SQL_QUOTE_SHOW_CREATE = 1/*!40101, @@SQL_MODE='NO_AUTO_VALUE_ON_ZERO,STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION'*/ 1052 Query SELECT @@server_id /*!50038 , @@hostname*/
1053 Connect root@localhost on test 1053 Query set autocommit=1 1053 Query SHOW VARIABLES LIKE 'innodb\_lock_wait_timeout' 1053 Query SET SESSION innodb_lock_wait_timeout=1 1053 Query SHOW VARIABLES LIKE 'lock\_wait_timeout' 1053 Query SET SESSION lock_wait_timeout=60 1053 Query SHOW VARIABLES LIKE 'wait\_timeout' 1053 Query SET SESSION wait_timeout=10000 1053 Query SELECT @@SQL_MODE 1053 Query SET @@SQL_QUOTE_SHOW_CREATE = 1/*!40101, @@SQL_MODE='NO_AUTO_VALUE_ON_ZERO,STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION'*/ 1053 Query SELECT @@server_id /*!50038 , @@hostname*/
上述主要是設置會話的變量信息，包括innodb_lock_wait_timeout，wait_timeout和SQL_QUOTE_SHOW_CREATE。
1052 Query SHOW VARIABLES LIKE 'wsrep_on' 1052 Query SHOW VARIABLES LIKE 'version%' 1052 Query SHOW ENGINES 1052 Query SHOW VARIABLES LIKE 'innodb_version' 1052 Query SHOW VARIABLES LIKE 'innodb_stats_persistent' 1052 Query SELECT @@SERVER_ID 1052 Query SHOW GRANTS FOR CURRENT_USER() 1052 Query SHOW FULL PROCESSLIST 1052 Query SHOW SLAVE HOSTS 1052 Query SHOW GLOBAL STATUS LIKE 'Threads_running' 1052 Query SHOW GLOBAL STATUS LIKE 'Threads_running' 1052 Query SELECT CONCAT(@@hostname, @@port) 1052 Query SHOW TABLES FROM `test` LIKE 't2' 1052 Query SHOW TRIGGERS FROM `test` LIKE 't2' 1052 Query /*!40101 SET @OLD_SQL_MODE := @@SQL_MODE, @@SQL_MODE := '', @OLD_QUOTE := @@SQL_QUOTE_SHOW_CREATE, @@SQL_QUOTE_SHOW_CREATE := 1 */ 1052 Query USE `test` 1052 Query SHOW CREATE TABLE `test`.`t2` 1052 Query /*!40101 SET @@SQL_MODE := @OLD_SQL_MODE, @@SQL_QUOTE_SHOW_CREATE := @OLD_QUOTE */ 1052 Query EXPLAIN SELECT * FROM `test`.`t2` WHERE 1=1 1052 Query SELECT table_schema, table_name FROM information_schema.key_column_usage WHERE referenced_table_schema='test' AND referenced_table_name='t2' 1052 Query SHOW VARIABLES LIKE 'wsrep_on' 1052 Query /*!40101 SET @OLD_SQL_MODE := @@SQL_MODE, @@SQL_MODE := '', @OLD_QUOTE := @@SQL_QUOTE_SHOW_CREATE, @@SQL_QUOTE_SHOW_CREATE := 1 */
解釋：
1. 查看參數變量，當前用戶的權限，slave的信息，會話變量
2. 確認t2是否存在，t2上是否有觸發器
3. 查看執行計划
4. 查看是否t2表是否被其它表外鍵關聯。
39 Query USE `test` 39 Query SHOW CREATE TABLE `test`.`t2` 39 Query /*!40101 SET @@SQL_MODE := @OLD_SQL_MODE, @@SQL_QUOTE_SHOW_CREATE := @OLD_QUOTE */ 39 Query CREATE TABLE `test`.`_t2_new` ( `id` int(11) NOT NULL AUTO_INCREMENT, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=1005764 DEFAULT CHARSET=utf8 161121 12:49:18 39 Query ALTER TABLE `test`.`_t2_new` ADD COLUMN c1 DATETIME 39 Query /*!40101 SET @OLD_SQL_MODE := @@SQL_MODE, @@SQL_MODE := '', @OLD_QUOTE := @@SQL_QUOTE_SHOW_CREATE, @@SQL_QUOTE_SHOW_CREATE := 1 */ 39 Query USE `test` 39 Query SHOW CREATE TABLE `test`.`_t2_new` 39 Query /*!40101 SET @@SQL_MODE := @OLD_SQL_MODE, @@SQL_QUOTE_SHOW_CREATE := @OLD_QUOTE */ 39 Query CREATE TRIGGER `pt_osc_test_t2_del` AFTER DELETE ON `test`.`t2` FOR EACH ROW DELETE IGNORE FROM `test`.`_t2_new` WHERE `test`.`_t2_new`.`id` <=> OLD.`id` 39 Query CREATE TRIGGER `pt_osc_test_t2_upd` AFTER UPDATE ON `test`.`t2` FOR EACH ROW REPLACE INTO `test`.`_t2_new` (`id`) VALUES (NEW.`id`) 39 Query CREATE TRIGGER `pt_osc_test_t2_ins` AFTER INSERT ON `test`.`t2` FOR EACH ROW REPLACE INTO `test`.`_t2_new` (`id`) VALUES (NEW.`id`)
解釋：
1. 根據目標表結構創建一張新表。
2. 對新表添加字段，可以看出pt-online-shema-change對表結構進行變更依賴的還是MySQL自身的Online DDL。
3. 針對目標表創建三個觸發器，DELETE，UPDATE和INSERT，因為REPLACE操作只有在主鍵或唯一索引存在的情況下才有意義，這也就解釋了為什么目標表上要有主鍵或唯一索引。
39 Query EXPLAIN SELECT * FROM `test`.`t2` WHERE 1=1 39 Query SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) ORDER BY `id` LIMIT 1 /*first lower boundary*/ 39 Query SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`t2` FORCE INDEX (`PRIMARY`) WHERE `id` IS NOT NULL ORDER BY `id` LIMIT 1 /*key_len*/ 39 Query EXPLAIN SELECT /*!40001 SQL_NO_CACHE */ * FROM `test`.`t2` FORCE INDEX (`PRIMARY`) WHERE `id` >= '1' /*key_len*/ 39 Query EXPLAIN SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1')) ORDER BY `id` LIMIT 999, 2 /*next chunk boundary*/ 39 Query SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1')) ORDER BY `id` LIMIT 999, 2 /*next chunk boundary*/ 39 Query EXPLAIN SELECT `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1')) AND ((`id` <= '1000')) LOCK IN SHARE MODE /*explain pt-online-schema-change 2352 copy nibble*/ 39 Query INSERT LOW_PRIORITY IGNORE INTO `test`.`_t2_new` (`id`) SELECT `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1')) AND ((`id` <= '1000')) LOCK IN SHARE MODE /*pt-online-schema-change 2352 copy nibble*/ 39 Query SHOW WARNINGS 39 Query SELECT @@SERVER_ID 39 Query SHOW GRANTS FOR CURRENT_USER() 39 Query SHOW FULL PROCESSLIST 39 Query SELECT @@SERVER_ID 39 Query SHOW GRANTS FOR CURRENT_USER() 39 Query SHOW FULL PROCESSLIST 161121 12:49:20 39 Query SELECT 'pt-online-schema-change keepalive' 161121 12:49:21 39 Query SELECT @@SERVER_ID 39 Query SHOW GRANTS FOR CURRENT_USER() 39 Query SHOW FULL PROCESSLIST
39 Query SHOW GLOBAL STATUS LIKE 'Threads_running' 39 Query EXPLAIN SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1001')) ORDER BY `id` LIMIT 28516, 2 /*next chunk boundary*/ 39 Query SELECT /*!40001 SQL_NO_CACHE */ `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1001')) ORDER BY `id` LIMIT 28516, 2 /*next chunk boundary*/ 39 Query EXPLAIN SELECT `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1001')) AND ((`id` <= '29517')) LOCK IN SHARE MODE /*explain pt-online-schema-change 2352 copy nibble*/ 39 Query INSERT LOW_PRIORITY IGNORE INTO `test`.`_t2_new` (`id`) SELECT `id` FROM `test`.`t2` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '1001')) AND ((`id` <= '29517')) LOCK IN SHARE MODE /*pt-online-schema-change 2352 copy nibble*/
解釋：
上述輸出只包含兩個chunk的選擇。其它chunk的選擇基本相同。
1. SHOW GLOBAL STATUS LIKE 'Threads_running'用於監控當前的系統負載。
2. 可以看出pt-online-schema-change是以chunk為單位進行目標表數據的拷貝。
3. 在拷貝的過程中，對目標表的相關記錄加了共享鎖，此時，會堵塞客戶端對這些記錄的DML操作。
39 Query ANALYZE TABLE `test`.`_t2_new` /* pt-online-schema-change */ 161121 12:50:32 39 Query RENAME TABLE `test`.`t2` TO `test`.`_t2_old`, `test`.`_t2_new` TO `test`.`t2` 161121 12:50:35 39 Query DROP TABLE IF EXISTS `test`.`_t2_old` 161121 12:50:36 39 Query DROP TRIGGER IF EXISTS `test`.`pt_osc_test_t2_del` 39 Query DROP TRIGGER IF EXISTS `test`.`pt_osc_test_t2_upd` 39 Query DROP TRIGGER IF EXISTS `test`.`pt_osc_test_t2_ins` 39 Query SHOW TABLES FROM `test` LIKE '\_t2\_new' 161121 12:50:37 40 Quit 39 Quit
解釋：
1. 在完成數據的拷貝后，會對新表執行ANALYZE操作，這樣，可及時更新新表的統計信息。
官檔的解釋如下：
This circumvents a potentially serious issue related to InnoDB optimizer statistics. If the table being alerted is busy and the tool completes quickly, the new table will not have optimizer statistics after being swapped. This can cause fast, index-using queries to do full table scans until optimizer statistics are updated (usually after 10 seconds). If the table is large and the server very busy, this can cause an outage.
2. 對目標表和新表進行RENAME操作。
3. 刪除原來的目標表
4. 刪除觸發器。