ProxySQL 排錯
- 問題分析:
在ProxySQL在集群下,因未知原因導致誤測到所有節點OFFLINE_HARD,並runtime_mysql_servers表清空,從而導致前端查詢無法傳遞到后端對應的servers上,重新load mysql servers to runtime可以暫時解決,但沒有找到長期解決辦法,只好退回單節點模式。如有解決的兄弟希望你分享給我(k2865#qq.com,或討論區留言 ) - 版本和環境
ProxySQL:
admin@127.0.0.1 [(none)]> select @@Version;
+--------------------+
| @@version |
+--------------------+
| v1.4.10-1-g5eb0f3e |
+--------------------+
1 row in set (0.00 sec)
admin@127.0.0.1 [(none)]>select * from disk.proxysql_servers;
+----------+------+--------+-----------+
| hostname | port | weight | comment |
+----------+------+--------+-----------+
| db211 | 6032 | 1 | secondary |
| db210 | 6032 | 1 | primary |
+----------+------+--------+-----------+
2 rows in set (0.00 sec)
--
--
MGR:
root@localhost [sysbench_testdata]>select version();
+------------+
| version() |
+------------+
| 5.7.23-log |
+------------+
1 row in set (0.00 sec)
root@localhost [sysbench_testdata]>select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | 33a6c4f8-a67a-11e8-b7f6-0050568beaf6 | db211 | 3507 | ONLINE |
| group_replication_applier | 951678d2-91a2-11e8-be71-0050568be82c | db210 | 3507 | ONLINE |
| group_replication_applier | abac4f08-a683-11e8-9447-0050568b744a | db212 | 3507 | ONLINE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
3 rows in set (0.00 sec)
- 故障現像:
ProxySQL 開始最先是正常的.可以讀寫,通過sysbench 也能正常壓測,但隨后報以下錯誤:
FATAL: MySQL error: 9001 "Max connect timeout reached while reaching hostgroup 10 after 10000ms"
(last message repeated 1 times)
FATAL: `thread_init' function failed: /usr/share/sysbench/oltp_common.lua:273: SQL API error
(last message repeated 1 times)
FATAL: mysql_stmt_prepare() failed
FATAL: MySQL error: 9001 "Max connect timeout reached while reaching hostgroup 10 after 10000ms"
FATAL: mysql_stmt_prepare() failed
FATAL: MySQL error: 9001 "Max connect timeout reached while reaching hostgroup 10 after 10000ms"
FATAL: mysql_stmt_prepare() failed
FATAL: MySQL error: 9001 "Max connect timeout reached while reaching hostgroup 10 after 10001ms"
FATAL: mysql_stmt_prepare() failed
- 手工連接,發現直接連接MGR節點是可以正常連接並查詢的,連接proxysql,可以登錄但無法查詢:
wyz@db210 [(none)]>select * from wenyz.t2;
ERROR 9001 (HY000): Max connect timeout reached while reaching hostgroup 11 after 10000ms
wyz@db210 [(none)]>select * from wenyz.t2;
ERROR 9001 (HY000): Max connect timeout reached while reaching hostgroup 11 after 10000ms
查看官方問題庫,得以下解釋:
11. No Hostgroup 0 has been configured , then why we get "Max connect timeout reached while reaching hostgroup 0 after 10000ms" ?
This is how MySQL Query Rules work while selecting hostgroups.
When you set any query rules inside table mysql_query_rules , Then your query get analyzed by the Query Processor to decide which destination hostgroup it should forward. (according to mysql_query_rules.destination_hostgroup)
When your Query Processor doesn't find any query matching to query rule then the default hostgroup for the specific user is applied (according to mysql_users.default_hostgroup)
Example : Default Hostgroup for User
根據這一信息,查看ProxySQL狀態,發現runtime_mysql_servers沒有記錄.
admin@127.0.0.1 [(none)]>select * from mysql_servers;
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 10 | db212 | 3507 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 11 | db210 | 3507 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 11 | db212 | 3507 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 11 | db211 | 3507 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 13 | db211 | 3507 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 13 | db210 | 3507 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
6 rows in set (0.00 sec)
admin@127.0.0.1 [(none)]>select * from runtime_mysql_servers;
Empty set (0.00 sec)
admin@127.0.0.1 [(none)]>
- 重新執行load mysql servers to runtime后恢復正常
admin@127.0.0.1 [(none)]>load mysql servers to runtime;
Query OK, 0 rows affected (0.00 sec)
- 問題重現時proxysql.log中出現以下信息:
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:308 , MyDS:308) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:311 , MyDS:311) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:312 , MyDS:312) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:315 , MyDS:315) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:320 , MyDS:320) , 1040: Too many connections.
2018-08-25 23:33:06 MySQL_HostGroups_Manager.cpp:303:connect_error(): [ERROR] Shunning server db212:3507 with 5 errors/sec. Shunning for 10 seconds
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:322 , MyDS:322) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:309 , MyDS:309) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:326 , MyDS:326) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:313 , MyDS:313) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:328 , MyDS:328) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:317 , MyDS:317) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:325 , MyDS:325) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:334 , MyDS:334) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:332 , MyDS:332) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:331 , MyDS:331) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:307 , MyDS:307) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:310 , MyDS:310) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:314 , MyDS:314) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:316 , MyDS:316) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:318 , MyDS:318) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:321 , MyDS:321) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:324 , MyDS:324) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:327 , MyDS:327) , 1040: Too many connections.
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:330 , MyDS:330) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:333 , MyDS:333) , 1040: Too many connections.
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:335 , MyDS:335) , 1040: Too many connections.
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:336 , MyDS:336) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:337 , MyDS:337) , 1040: Too many connections.
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: SHUNNED , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-25 23:33:06 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 1 | 0 | 1000 | 0 | 0 | 0 | | 140420659150080 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659545088 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659544960 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659544832 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548032 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549696 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-25 23:33:11 MySQL_HostGroups_Manager.cpp:2580:update_group_replication_set_writer(): [WARNING] Group Replication: setting host db211:3507 as writer
2018-08-25 23:33:11 MySQL_HostGroups_Manager.cpp:2580:update_group_replication_set_writer(): [WARNING] Group Replication: setting host db210:3507 as writer
2018-08-25 23:33:11 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: SHUNNED , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-25 23:33:11 [INFO] Dumping mysql_servers
- 出現以上信息時,sysbench端報ERROR 9001 (HY000): Max connect timeout reached while reaching hostgroup 11 after 10000ms了.但proxysql的runtime_mysql_server中還是正常的.在proxysql.log中有大量的deadlock 或lock timeout,輸出的runtime_mysql_server信息是全的.但到第二天早上才有如下:
2018-08-26 08:16:27 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 08:16:27 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659150080 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659545088 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420789185920 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659544832 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548032 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549696 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 08:16:27 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 08:16:27 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659150080 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659545088 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420789185920 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659544832 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548032 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549696 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 08:16:27 [INFO] Dumping mysql_servers_incoming
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2018-08-26 08:16:27 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659150080, hostgroup 10, address db212 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 08:16:27 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659545088, hostgroup 13, address db210 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 08:16:27 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420789185920, hostgroup 13, address db211 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 08:16:27 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659544832, hostgroup 11, address db211 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 08:16:27 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659548032, hostgroup 11, address db212 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 08:16:27 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659549696, hostgroup 11, address db210 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 08:16:27 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 08:16:27 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659150080 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659545088 |
| 13 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420789185920 |
| 11 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659544832 |
| 11 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548032 |
| 11 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549696 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 08:16:27 [INFO] Dumping current MySQL Servers structures for hostgroup 10
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 08:16:27 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659150080 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 [INFO] Dumping current MySQL Servers structures for hostgroup 13
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 08:16:27 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659150080 |
| 13 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420789185920 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659545088 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 08:16:27 [INFO] Dumping current MySQL Servers structures for hostgroup 11
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 08:16:27 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659150080 |
| 13 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420789185920 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659545088 |
| 11 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549696 |
| 11 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659544832 |
| 11 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548032 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 08:16:27 [INFO] Dumping current MySQL Servers structures for hostgroup 15
2018-08-26 08:16:27 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659150080 |
| 13 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420789185920 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659545088 |
| 11 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549696 |
| 11 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659544832 |
| 11 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548032 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:28 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 08:16:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 11 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659544832 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659545088 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 08:16:28 [INFO] Cluster: detected a new checksum for mysql_servers from peer db210:6032, version 16925, epoch 1535242587, checksum 0xEA6375A74FCB9E77 . Not syncing yet ...
2018-08-26 08:16:28 [INFO] Cluster: checksum for mysql_servers from peer db210:6032 matches with local checksum 0xEA6375A74FCB9E77 , we won't sync.
2018-08-26 08:16:30 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
2018-08-26 08:16:30 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
\\\ 另一次故障日志
2018-08-26 12:22:23 [INFO] Dumping current MySQL Servers structures for hostgroup 10
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:23 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:23 [INFO] Dumping current MySQL Servers structures for hostgroup 13
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:23 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:23 [INFO] Dumping current MySQL Servers structures for hostgroup 11
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:23 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:23 [INFO] Dumping current MySQL Servers structures for hostgroup 15
2018-08-26 12:22:23 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:24 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:24 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 MySQL_HostGroups_Manager.cpp:2580:update_group_replication_set_writer(): [WARNING] Group Replication: setting host db211:3507 as writer
2018-08-26 12:22:28 [INFO] Group replication: max_writers=1 , moving 1 nodes from writer HG 10 to backup HG 13
2018-08-26 12:22:28 MySQL_HostGroups_Manager.cpp:2580:update_group_replication_set_writer(): [WARNING] Group Replication: setting host db210:3507 as writer
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping mysql_servers_incoming
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup 10
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup 13
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup 11
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup 15
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping mysql_servers_incoming
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2018-08-26 12:22:28 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659549056, hostgroup 10, address db212 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 12:22:28 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659549184, hostgroup 13, address db210 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 12:22:28 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659549568, hostgroup 13, address db211 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 12:22:28 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659548800, hostgroup 11, address db211 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 12:22:28 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659548672, hostgroup 11, address db212 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 12:22:28 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659548544, hostgroup 11, address db210 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 13 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 11 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
| 11 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup 10
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup 13
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup 11
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 11 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
| 11 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup 15
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 11 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
| 11 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:29 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:29 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 11 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:29 [INFO] Cluster: detected a new checksum for mysql_servers from peer db210:6032, version 17848, epoch 1535257348, checksum 0xEA6375A74FCB9E77 . Not syncing yet ...
2018-08-26 12:22:29 [INFO] Cluster: checksum for mysql_servers from peer db210:6032 matches with local checksum 0xEA6375A74FCB9E77 , we won't sync.
2018-08-26 12:22:32 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
2018-08-26 12:22:32 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
2018-08-26 12:22:32 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
2018-08-26 12:22:32 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
2018-08-26 12:22:33 [INFO] Cluster: detected a new checksum for mysql_servers from peer db211:6032, version 14580, epoch 1535257352, checksum 0xEA6375A74FCB9E77 . Not syncing yet ...
2018-08-26 12:22:33 [INFO] Cluster: checksum for mysql_servers from peer db211:6032 matches with local checksum 0xEA6375A74FCB9E77 , we won't sync.
查看 集群狀態 發現沒有監測到runtime的變化:
admin@127.0.0.1 [(none)]>select *,from_unixtime(changed_at),from_unixtime(updated_at) from stats_proxysql_servers_checksums;
+----------+------+-------------------+---------+------------+--------------------+------------+------------+------------+---------------------------+---------------------------+
| hostname | port | name | version | epoch | checksum | changed_at | updated_at | diff_check | from_unixtime(changed_at) | from_unixtime(updated_at) |
+----------+------+-------------------+---------+------------+--------------------+------------+------------+------------+---------------------------+---------------------------+
| db211 | 6032 | admin_variables | 0 | 0 | | 0 | 1535258863 | 0 | 1970-01-01 00:00:00 | 2018-08-26 04:47:43 |
| db211 | 6032 | mysql_query_rules | 2 | 1535111193 | 0xC2127DA38220E59D | 1535174701 | 1535258863 | 0 | 2018-08-25 05:25:01 | 2018-08-26 04:47:43 |
| db211 | 6032 | mysql_servers | 14580 | 1535257352 | 0xEA6375A74FCB9E77 | 1535257353 | 1535258863 | 0 | 2018-08-26 04:22:33 | 2018-08-26 04:47:43 |
| db211 | 6032 | mysql_users | 2 | 1535105812 | 0x6C619C922DBF30FF | 1535174701 | 1535258863 | 0 | 2018-08-25 05:25:01 | 2018-08-26 04:47:43 |
| db211 | 6032 | mysql_variables | 0 | 0 | | 0 | 1535258863 | 0 | 1970-01-01 00:00:00 | 2018-08-26 04:47:43 |
| db211 | 6032 | proxysql_servers | 8 | 1535125307 | 0x53CAAA0FF8478223 | 1535174701 | 1535258863 | 0 | 2018-08-25 05:25:01 | 2018-08-26 04:47:43 |
| db210 | 6032 | admin_variables | 0 | 0 | | 0 | 1535258863 | 0 | 1970-01-01 00:00:00 | 2018-08-26 04:47:43 |
| db210 | 6032 | mysql_query_rules | 1 | 1535174701 | 0xC2127DA38220E59D | 1535174702 | 1535258863 | 0 | 2018-08-25 05:25:02 | 2018-08-26 04:47:43 |
| db210 | 6032 | mysql_servers | 17848 | 1535257348 | 0xEA6375A74FCB9E77 | 1535257349 | 1535258863 | 0 | 2018-08-26 04:22:29 | 2018-08-26 04:47:43 |
| db210 | 6032 | mysql_users | 1 | 1535174701 | 0x6C619C922DBF30FF | 1535174702 | 1535258863 | 0 | 2018-08-25 05:25:02 | 2018-08-26 04:47:43 |
| db210 | 6032 | mysql_variables | 0 | 0 | | 0 | 1535258863 | 0 | 1970-01-01 00:00:00 | 2018-08-26 04:47:43 |
| db210 | 6032 | proxysql_servers | 1 | 1535174701 | 0x53CAAA0FF8478223 | 1535174702 | 1535258863 | 0 | 2018-08-25 05:25:02 | 2018-08-26 04:47:43 |
+----------+------+-------------------+---------+------------+--------------------+------------+------------+------------+---------------------------+---------------------------+
12 rows in set (0.00 sec)
ProxySQL全配置信息
admin@127.0.0.1 [(none)]>show global variables;
+-----------------------------------------------------+--------------------------------+
| Variable_name | Value |
+-----------------------------------------------------+--------------------------------+
| admin-admin_credentials | admin:admin;cluster_21x:zstxxx |
| admin-checksum_mysql_query_rules | true |
| admin-checksum_mysql_servers | true |
| admin-checksum_mysql_users | true |
| admin-cluster_check_interval_ms | 1000 |
| admin-cluster_check_status_frequency | 10 |
| admin-cluster_mysql_query_rules_diffs_before_sync | 3 |
| admin-cluster_mysql_query_rules_save_to_disk | true |
| admin-cluster_mysql_servers_diffs_before_sync | 3 |
| admin-cluster_mysql_servers_save_to_disk | true |
| admin-cluster_mysql_users_diffs_before_sync | 3 |
| admin-cluster_mysql_users_save_to_disk | true |
| admin-cluster_password | zstxxx |
| admin-cluster_proxysql_servers_diffs_before_sync | 3 |
| admin-cluster_proxysql_servers_save_to_disk | true |
| admin-cluster_username | cluster_21x |
| admin-hash_passwords | true |
| admin-mysql_ifaces | 0.0.0.0:6032 |
| admin-read_only | false |
| admin-refresh_interval | 2000 |
| admin-stats_credentials | stats:stats |
| admin-stats_mysql_connection_pool | 60 |
| admin-stats_mysql_connections | 60 |
| admin-stats_mysql_query_cache | 60 |
| admin-stats_system_cpu | 60 |
| admin-stats_system_memory | 60 |
| admin-telnet_admin_ifaces | (null) |
| admin-telnet_stats_ifaces | (null) |
| admin-version | v1.4.10-1-g5eb0f3e |
| admin-web_enabled | false |
| admin-web_port | 6080 |
| mysql-autocommit_false_is_transaction | false |
| mysql-autocommit_false_not_reusable | false |
| mysql-client_found_rows | true |
| mysql-client_multi_statements | true |
| mysql-commands_stats | true |
| mysql-connect_retries_delay | 1 |
| mysql-connect_retries_on_failure | 10 |
| mysql-connect_timeout_server | 3000 |
| mysql-connect_timeout_server_max | 10000 |
| mysql-connection_delay_multiplex_ms | 0 |
| mysql-connection_max_age_ms | 0 |
| mysql-connpoll_reset_queue_length | 50 |
| mysql-default_charset | utf8 |
| mysql-default_max_latency_ms | 1000 |
| mysql-default_query_delay | 0 |
| mysql-default_query_timeout | 36000000 |
| mysql-default_reconnect | true |
| mysql-default_schema | information_schema |
| mysql-default_sql_mode | |
| mysql-default_time_zone | SYSTEM |
| mysql-enforce_autocommit_on_reads | false |
| mysql-eventslog_filename | |
| mysql-eventslog_filesize | 104857600 |
| mysql-forward_autocommit | false |
| mysql-free_connections_pct | 10 |
| mysql-have_compress | true |
| mysql-hostgroup_manager_verbose | 1 |
| mysql-init_connect | |
| mysql-interfaces | 0.0.0.0:6033 |
| mysql-kill_backend_connection_when_disconnect | true |
| mysql-long_query_time | 1000 |
| mysql-max_allowed_packet | 4194304 |
| mysql-max_connections | 2048 |
| mysql-max_stmts_cache | 10000 |
| mysql-max_stmts_per_connection | 20 |
| mysql-max_transaction_time | 14400000 |
| mysql-mirror_max_concurrency | 16 |
| mysql-mirror_max_queue_length | 32000 |
| mysql-monitor_connect_interval | 60000 |
| mysql-monitor_connect_timeout | 600 |
| mysql-monitor_enabled | true |
| mysql-monitor_groupreplication_healthcheck_interval | 5000 |
| mysql-monitor_groupreplication_healthcheck_timeout | 800 |
| mysql-monitor_history | 600000 |
| mysql-monitor_password | zstzst |
| mysql-monitor_ping_interval | 10000 |
| mysql-monitor_ping_max_failures | 3 |
| mysql-monitor_ping_timeout | 1000 |
| mysql-monitor_query_interval | 60000 |
| mysql-monitor_query_timeout | 100 |
| mysql-monitor_read_only_interval | 1500 |
| mysql-monitor_read_only_max_timeout_count | 3 |
| mysql-monitor_read_only_timeout | 500 |
| mysql-monitor_replication_lag_interval | 10000 |
| mysql-monitor_replication_lag_timeout | 1000 |
| mysql-monitor_replication_lag_use_percona_heartbeat | |
| mysql-monitor_slave_lag_when_null | 60 |
| mysql-monitor_username | proxysql |
| mysql-monitor_wait_timeout | true |
| mysql-monitor_writer_is_also_reader | true |
| mysql-multiplexing | true |
| mysql-ping_interval_server_msec | 120000 |
| mysql-ping_timeout_server | 500 |
| mysql-poll_timeout | 2000 |
| mysql-poll_timeout_on_failure | 100 |
| mysql-query_cache_size_MB | 256 |
| mysql-query_digests | true |
| mysql-query_digests_lowercase | false |
| mysql-query_digests_max_digest_length | 2048 |
| mysql-query_digests_max_query_length | 65000 |
| mysql-query_processor_iterations | 0 |
| mysql-query_processor_regex | 1 |
| mysql-query_retries_on_failure | 1 |
| mysql-server_capabilities | 45578 |
| mysql-server_version | 5.5.30 |
| mysql-servers_stats | true |
| mysql-session_idle_ms | 1000 |
| mysql-session_idle_show_processlist | true |
| mysql-sessions_sort | true |
| mysql-shun_on_failures | 5 |
| mysql-shun_recovery_time_sec | 10 |
| mysql-ssl_p2s_ca | |
| mysql-ssl_p2s_cert | |
| mysql-ssl_p2s_cipher | |
| mysql-ssl_p2s_key | |
| mysql-stacksize | 1048576 |
| mysql-stats_time_backend_query | false |
| mysql-stats_time_query_processor | false |
| mysql-threads | 4 |
| mysql-threshold_query_length | 524288 |
| mysql-threshold_resultset_size | 4194304 |
| mysql-throttle_connections_per_sec_to_hostgroup | 1000000 |
| mysql-throttle_max_bytes_per_second_to_client | 2147483647 |
| mysql-throttle_ratio_server_to_client | 0 |
| mysql-verbose_query_error | false |
| mysql-wait_timeout | 28800000 |
+-----------------------------------------------------+--------------------------------+
127 rows in set (0.00 sec)
最后ProxySQL切換成單節點,運行正常