某天根據領導的需求，搭建主從。備份的時候在上午，主從搭建是下午。因為搭建完，沒有進行主從校驗。導致主從數據不一致，然后引發了主從復制中斷。
show slave status\G; Last_SQL_Errno: 1032 Last_SQL_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 0 failed executing transaction '8a0446c4-cce4-11e5-865d-002564fbe7a0:540640886' at master log mysql-bin.000717, end_log_pos 925167524. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any. Replicate_Ignore_Server_Ids: Master_Server_Id: 44 Master_UUID: 8a0446c4-cce4-11e5-865d-002564fbe7a0 Master_Info_File: mysql.slave_master_info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: 180427 17:08:21 Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: 8a0446c4-cce4-11e5-865d-002564fbe7a0:540311459:540311461:540311467-540311470:540311476-540311478:540311496-540311498:540311504:540311510-541041993 Executed_Gtid_Set: 4a7292e6-49f7-11e8-b161-f04da23eacda:1-3187, 8a0446c4-cce4-11e5-865d-002564fbe7a0:1-540640885:540640954:540640956:540641046:540641052:540641121:540641128:540641132-540641133:540641147:540641255:540641261:540641269:540641299:540641329
查看error日志
[root@gfs1 free]# tail -n 100 /home/mysql3307/log/mysqld.log 2018-04-27T09:00:02.219598Z 17 [ERROR] Slave SQL for channel '': Worker 1 failed executing transaction '8a0446c4-cce4-11e5-865d-002564fbe7a0:540311459' at master log mysql-bin.000717, end_log_pos 602665891; Could not execute Write_rows event on table hp_cougar_transfer.BI_AppointmentOrder; Duplicate entry '89677392' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 602665891, Error_code: 1062 2018-04-27T09:00:02.219734Z 15 [Warning] Slave SQL for channel '': Slave worker thread has failed to apply an event. As a consequence, the coordinator thread is stopping execution. Error_code: 0 2018-04-27T09:00:02.219773Z 15 [Warning] Slave SQL for channel '': ... The slave coordinator and worker threads are stopped, possibly leaving data in inconsistent state. A restart should restore consistency automatically, although using non-transactional storage for data or info tables or DDL queries could lead to problems. In such cases you have to examine your data (see documentation for details). Error_code: 1756 2018-04-27T09:00:19.368059Z 14 [ERROR] Error reading packet from server for channel '': Lost connection to MySQL server during query (server_errno=2013) 2018-04-27T09:00:19.368102Z 14 [Note] Slave I/O thread killed while reading event for channel '' 2018-04-27T09:00:19.368114Z 14 [Note] Slave I/O thread exiting for channel '', read up to log 'mysql-bin.000717', position 923581274 2018-04-27T09:00:28.997607Z 13 [Note] 'CHANGE MASTER TO FOR CHANNEL '' executed'. Previous state master_host='20.1.31.44', master_port= 3306, master_log_file='mysql-bin.000717', master_log_pos= 923581274, master_bind=''. New state master_host='20.1.31.44', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''. 2018-04-27T09:00:33.576115Z 18 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. 2018-04-27T09:00:33.579294Z 19 [Note] Slave SQL thread for channel '' initialized, starting replication in log 'FIRST' at position 0, relay log '/home/mysql3307/relay/relay-bin.000001' position: 4 2018-04-27T09:00:33.583110Z 18 [Note] Slave I/O thread for channel '': connected to master 'repl@20.1.31.44:3306',replication started in log 'FIRST' at position 4 2018-04-27T09:00:37.154304Z 21 [ERROR] Slave SQL for channel '': Worker 1 failed executing transaction '8a0446c4-cce4-11e5-865d-002564fbe7a0:540311459' at master log , end_log_pos 602665891; Could not execute Write_rows event on table hp_cougar_transfer.BI_AppointmentOrder; Duplicate entry '89677392' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 602665891, Error_code: 1062 2018-04-27T09:00:37.155089Z 19 [Warning] Slave SQL for channel '': ... The slave coordinator and worker threads are stopped, possibly leaving data in inconsistent state. A restart should restore consistency automatically, although using non-transactional storage for data or info tables or DDL queries could lead to problems. In such cases you have to examine your data (see documentation for details). Error_code: 1756 2018-04-27T09:00:37.155143Z 19 [Warning] Slave SQL for channel '': ... The slave coordinator and worker threads are stopped, possibly leaving data in inconsistent state. A restart should restore consistency automatically, although using non-transactional storage for data or info tables or DDL queries could lead to problems. In such cases you have to examine your data (see documentation for details). Error_code: 1756 2018-04-27T09:06:26.445883Z 13 [Note] @@GLOBAL.GTID_PURGED was changed from '' to '8a0446c4-cce4-11e5-865d-002564fbe7a0:1-540640764'. 2018-04-27T09:06:26.445922Z 13 [Note] @@GLOBAL.GTID_EXECUTED was changed from '' to '8a0446c4-cce4-11e5-865d-002564fbe7a0:1-540640764'. 2018-04-27T09:06:30.593608Z 22 [Note] Slave SQL thread for channel '' initialized, starting replication in log 'mysql-bin.000717' at position 602663272, relay log '/home/mysql3307/relay/relay-bin.000002' position: 414 2018-04-27T09:08:21.784025Z 23 [ERROR] Slave SQL for channel '': Worker 0 failed executing transaction '8a0446c4-cce4-11e5-865d-002564fbe7a0:540640886' at master log mysql-bin.000717, end_log_pos 925167524; Could not execute Update_rows event on table hp_cougar_transfer.BI_BlackUser; Can't find record in 'BI_BlackUser', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.000717, end_log_pos 925167524, Error_code: 1032 2018-04-27T09:08:21.792219Z 22 [Warning] Slave SQL for channel '': ... The slave coordinator and worker threads are stopped, possibly leaving data in inconsistent state. A restart should restore consistency automatically, although using non-transactional storage for data or info tables or DDL queries could lead to problems. In such cases you have to examine your data (see documentation for details). Error_code: 1756
發現在17:08:21的時候，主庫進行了update操作
Could not execute Update_rows event on table hp_cougar_transfer.BI_BlackUser; Can't find record in 'BI_BlackUser', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND;
原因是：
在master上更新一條記錄，而slave上找不到，丟失了數據
[root@dbs2 sa]# mysqlbinlog -uroot -pHdkj@idc3 /home/mysql57/log/mysql-bin.000717 --base64-output=decode-rows -v -v --start-datetime='2018-4-27 17:04:05' |grep -A '17' 925167524 >/home/free/error.log
[root@gfs1 free]# cat error.log #180427 17:04:28 server id 44 end_log_pos 925167524 CRC32 0x894bd673 Update_rows: table id 19759 flags: STMT_END_F ### UPDATE `hp_cougar_transfer`.`BI_BlackUser` ### WHERE ### @1=16702124 /* INT meta=0 nullable=0 is_null=0 */ ### @2=233197302 /* INT meta=0 nullable=0 is_null=0 */ ### @3='余王翠' /* VARSTRING(300) meta=300 nullable=0 is_null=0 */ ### @4='330325196807144943' /* VARSTRING(60) meta=60 nullable=0 is_null=0 */ ### @5='13454062388' /* VARSTRING(45) meta=45 nullable=0 is_null=0 */ ### @6=2018 /* INT meta=0 nullable=0 is_null=0 */ ### @7=2 /* INT meta=0 nullable=0 is_null=0 */ ### @8=1 /* INT meta=0 nullable=0 is_null=0 */ ### @9=0 /* INT meta=0 nullable=0 is_null=0 */ ### @10=2 /* TINYINT meta=0 nullable=0 is_null=0 */ ### @11='' /* VARSTRING(300) meta=300 nullable=0 is_null=0 */ ### @12=1524819613 /* TIMESTAMP(0) meta=0 nullable=0 is_null=0 */ ### SET ### @1=16702124 /* INT meta=0 nullable=0 is_null=0 */ ### @2=233197302 /* INT meta=0 nullable=0 is_null=0 */ -- # at 925167524 #180427 17:04:28 server id 44 end_log_pos 925167555 CRC32 0x2b813b59 Xid = 3408140482 COMMIT/*!*/; # at 925167555 #180427 17:04:29 server id 44 end_log_pos 925167620 CRC32 0x02d0730c GTID last_committed=1005465 sequence_number=1005466 SET @@SESSION.GTID_NEXT= '8a0446c4-cce4-11e5-865d-002564fbe7a0:540640887'/*!*/; # at 925167620 #180427 17:04:29 server id 44 end_log_pos 925167699 CRC32 0x0bf8daed Query thread_id=65361008 exec_time=0 error_code=0 SET TIMESTAMP=1524819869/*!*/; BEGIN /*!*/; # at 925167699 #180427 17:04:29 server id 44 end_log_pos 925167911 CRC32 0x0eab5376 Rows_query # INSERT INTO HL_LoginLog_BJGH(LoginName,IDNo,MobileNo,LoginIP,LoginStepType,LoginVerboseStatus,ServiceChannelType,`Status`) VALUES ('',null,'13261053197','20.1.31.40,41.78.243.233',3,3,8,2) # at 925167911 #180427 17:04:29 server id 44 end_log_pos 925167991 CRC32 0x0e9042e1 Table_map: `hl_log`.`HL_LoginLog_BJGH` mapped to number 19748 # at 925167991 #180427 17:04:29 server id 44 end_log_pos 925168086 CRC32 0xe932a3db Write_rows: table id 19748 flags: STMT_END_F
在slave上，查找下更新后的那條記錄，應該是不存在的。
把丟失的數據在slave上填補，然后跳過報錯即可。
mysql> insert into BI_BlackUser values (16702124,'233197302');
Query OK, 1 row affected (0.00 sec)