最近一MySQL服務器，由於一些特殊因素遇到“ERROR 1129 (00000): Host 'xxx' is blocked because of many connection errors. Unblock with 'mysqladmin flush-hosts'”，在問題解決后，在詳細了解參數max_connect_errors的過程中，有些不同網絡資料的矛盾描述確實讓我有點迷惑和混淆（關於這個錯誤，本質原因是因為同一個IP在短時間內產生太多中斷的數據庫連接（超過max_connect_errors的最大值）而導致的），下面介紹我的探索問題、分析問題、釋疑的一個過程。
首先，我在網上搜索了一些資料，不少資料信誓旦旦的介紹，密碼輸入錯誤的嘗試次數超過max_connect_errors變量，MySQL就會阻塞這個客戶端登錄，然后我找到了官方資料關於max_connect_errors的介紹，如下所示，MySQL 5.6/5.7的介紹一致
If more than this many successive connection requests from a host are interrupted without a successful connection, the server blocks that host from further connections. You can unblock blocked hosts by flushing the host cache. To do so, issue a FLUSH HOSTS statement or execute a mysqladmin flush-hosts command. If a connection is established successfully within fewer than max_connect_errors attempts after a previous connection was interrupted, the error count for the host is cleared to zero. However, once a host is blocked, flushing the host cache is the only way to unblock it. The default is 100.
如上所示，翻譯出來的話，大致如下：如果MySQL服務器連續接收到了來自於同一個主機的請求，而且這些連續的請求全部都沒有成功的建立連接就被中斷了，當這些連續的請求的累計值大於max_connect_errors的設定值時，MySQL服務器就會阻止這台主機后續的所有請求。相信一開始你看到這些資料，也會被“many successive connection requests from a host are interrupted without a successful connection”給弄懵，其實這個就是因為由於網絡異常而中止數據庫連接。網上搜索到這么一個資料：
There seems to be confusion around that variable. It does not really block hosts for repeated invalid passwords but for aborted connections due to network errors.
好吧，那么我們自己動手實驗驗證一下，就能弄明白到底那個是正確的。在MySQL數據庫里面創建一個test賬號，然后我們將max_connect_errors變量設置為3.
mysql> show variables like '%max_connect_errors%';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| max_connect_errors | 100 |
+--------------------+-------+
1 row in set (0.00 sec)
mysql> set global max_connect_errors=3;
Query OK, 0 rows affected (0.00 sec)
mysql> show variables like '%max_connect_error%';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| max_connect_errors | 3 |
+--------------------+-------+
1 row in set (0.00 sec)
然后我們在另外一台測試機器，以錯誤的密碼去連接這個MySQL數據庫，如下所示，即使前面輸入了三次錯誤密碼，第四次輸入是也沒有碰到上面錯誤。那么可以排除這個變量與密碼錯誤輸入有關系。
[root@mytestlnx02 tmp]# mysql -h10.20.57.24 -utest -p
Enter password:
ERROR 1045 (28000): Access denied for user 'test'@'mytestlnx02' (using password: YES)
[root@mytestlnx02 tmp]# mysql -h10.20.57.24 -utest -p
Enter password:
ERROR 1045 (28000): Access denied for user 'test'@'mytestlnx02' (using password: YES)
[root@mytestlnx02 tmp]# mysql -h10.20.57.24 -utest -p
Enter password:
ERROR 1045 (28000): Access denied for user 'test'@'mytestlnx02' (using password: YES)
[root@mytestlnx02 tmp]# mysql -h10.20.57.24 -utest -p
Enter password:
ERROR 1045 (28000): Access denied for user 'test'@'mytestlnx02' (using password: YES)
[root@mytestlnx02 tmp]#
其實，關於某個IP輸入了錯誤密碼，MySQL會在performance_schema數據庫下的host_cache表中記錄。它會累計記錄在COUNT_AUTHENTICATION_ERRORS字段，如下所示：
mysql> use performance_schema;
Database changed
mysql> select * from host_cache\G;
*************************** 1. row ***************************
IP: 192.168.27.180
HOST: gettestlnx02
HOST_VALIDATED: YES
SUM_CONNECT_ERRORS: 0
COUNT_HOST_BLOCKED_ERRORS: 0
COUNT_NAMEINFO_TRANSIENT_ERRORS: 0
COUNT_NAMEINFO_PERMANENT_ERRORS: 0
COUNT_FORMAT_ERRORS: 0
COUNT_ADDRINFO_TRANSIENT_ERRORS: 0
COUNT_ADDRINFO_PERMANENT_ERRORS: 0
COUNT_FCRDNS_ERRORS: 0
COUNT_HOST_ACL_ERRORS: 0
COUNT_NO_AUTH_PLUGIN_ERRORS: 0
COUNT_AUTH_PLUGIN_ERRORS: 0
COUNT_HANDSHAKE_ERRORS: 0
COUNT_PROXY_USER_ERRORS: 0
COUNT_PROXY_USER_ACL_ERRORS: 0
COUNT_AUTHENTICATION_ERRORS: 4
COUNT_SSL_ERRORS: 0
COUNT_MAX_USER_CONNECTIONS_ERRORS: 0
COUNT_MAX_USER_CONNECTIONS_PER_HOUR_ERRORS: 0
COUNT_DEFAULT_DATABASE_ERRORS: 0
COUNT_INIT_CONNECT_ERRORS: 0
COUNT_LOCAL_ERRORS: 0
COUNT_UNKNOWN_ERRORS: 0
FIRST_SEEN: 2018-01-31 16:28:19
LAST_SEEN: 2018-01-31 16:28:26
FIRST_ERROR_SEEN: 2018-01-31 16:28:19
LAST_ERROR_SEEN: 2018-01-31 16:28:26
1 row in set (0.00 sec)
ERROR:
No query specified
官方資料介紹，host_cache的字段是統計被視為“阻塞”的連接錯誤的數量（根據max_connect_errors系統變量進行評估）。 只計算協議握手錯誤，並且僅用於通過驗證的主機（HOST_VALIDATED = YES）。
SUM_CONNECT_ERRORS
The number of connection errors that are deemed “blocking” (assessed against the max_connect_errors system variable). Only protocol handshake errors are counted, and only for hosts that passed validation (HOST_VALIDATED = YES).
MySQL客戶端與數據庫建立連接需要發起三次握手協議，正常情況下，這個時間非常短，但是一旦網絡異常，網絡超時等因素出現，就會導致這個握手協議無法完成，MySQL有個參數connect_timeout，它是MySQL服務端進程mysqld等待連接建立完成的時間，單位為秒。如果超過connect_timeout時間范圍內，仍然無法完成協議握手話，MySQL客戶端會收到異常，異常消息類似於: Lost connection to MySQL server at 'XXX', system error: errno，該變量默認是10秒：
mysql> show variables like 'connect_timeout';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| connect_timeout | 10 |
+-----------------+-------+
1 row in set (0.00 sec)
mysql>
那么我們就構造一個網絡超時引起的數據庫連接被中斷案例吧，我們用Linux下的netem與tc命令模擬構造出復雜環境下的網絡傳輸延時案例，如下設置后，此時從測試服務器去訪問MySQL服務器，都會出現延時11秒：
[root@gettestlnx02 ~]# ping 10.20.57.24
PING 10.20.57.24 (10.20.57.24) 56(84) bytes of data.
64 bytes from 10.20.57.24: icmp_seq=1 ttl=62 time=0.251 ms
64 bytes from 10.20.57.24: icmp_seq=2 ttl=62 time=0.330 ms
64 bytes from 10.20.57.24: icmp_seq=3 ttl=62 time=0.362 ms
64 bytes from 10.20.57.24: icmp_seq=4 ttl=62 time=0.316 ms
64 bytes from 10.20.57.24: icmp_seq=5 ttl=62 time=0.281 ms
64 bytes from 10.20.57.24: icmp_seq=6 ttl=62 time=0.377 ms
^C
--- 10.20.57.24 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5716ms
rtt min/avg/max/mdev = 0.251/0.319/0.377/0.047 ms
[root@gettestlnx02 ~]# tc qdisc add dev eth0 root netem delay 11000ms
[root@gettestlnx02 ~]# ping 10.20.57.24
PING 10.20.57.24 (10.20.57.24) 56(84) bytes of data.
64 bytes from 10.20.57.24: icmp_seq=1 ttl=62 time=11000 ms
64 bytes from 10.20.57.24: icmp_seq=2 ttl=62 time=11000 ms
64 bytes from 10.20.57.24: icmp_seq=3 ttl=62 time=11000 ms
64 bytes from 10.20.57.24: icmp_seq=4 ttl=62 time=11000 ms
64 bytes from 10.20.57.24: icmp_seq=5 ttl=62 time=11000 ms
64 bytes from 10.20.57.24: icmp_seq=6 ttl=62 time=11000 ms
64 bytes from 10.20.57.24: icmp_seq=7 ttl=62 time=11000 ms
我們在測試服務器gettestlnx02連接MySQL數據庫，如下所示（注意，如果你是在通過ssh連接這台服務器的話，此時在gettestlnx02上操作會相當慢。當然你也可以在MySQL服務器模擬網絡延時，或者你將connect_timeout和網絡延時都設小一點）
[root@gettestlnx02 ~]# mysql -h10.20.57.24 -utest -p
Enter password:
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading authorization packet', system error: 0
[root@gettestlnx02 ~]#
如上所示，由於網絡延時超過10秒，導致連接MySQL失敗，此時，你在MySQL服務器上查詢host_cache表時,那么你就會看到SUM_CONNECT_ERRORS變成1了，COUNT_HANDSHAKE_ERRORS也變成了1.
那么我們反復這樣折騰三次，那么你會看到SUM_CONNECT_ERRORS變成3了，COUNT_HANDSHAKE_ERRORS也變成了3了.
mysql> select * from host_cache\G;
*************************** 1. row ***************************
IP: 192.168.27.180
HOST: gettestlnx02
HOST_VALIDATED: YES
SUM_CONNECT_ERRORS: 3
COUNT_HOST_BLOCKED_ERRORS: 1
COUNT_NAMEINFO_TRANSIENT_ERRORS: 0
COUNT_NAMEINFO_PERMANENT_ERRORS: 0
COUNT_FORMAT_ERRORS: 0
COUNT_ADDRINFO_TRANSIENT_ERRORS: 0
COUNT_ADDRINFO_PERMANENT_ERRORS: 0
COUNT_FCRDNS_ERRORS: 0
COUNT_HOST_ACL_ERRORS: 0
COUNT_NO_AUTH_PLUGIN_ERRORS: 0
COUNT_AUTH_PLUGIN_ERRORS: 0
COUNT_HANDSHAKE_ERRORS: 3
COUNT_PROXY_USER_ERRORS: 0
COUNT_PROXY_USER_ACL_ERRORS: 0
COUNT_AUTHENTICATION_ERRORS: 4
COUNT_SSL_ERRORS: 0
COUNT_MAX_USER_CONNECTIONS_ERRORS: 0
COUNT_MAX_USER_CONNECTIONS_PER_HOUR_ERRORS: 0
COUNT_DEFAULT_DATABASE_ERRORS: 0
COUNT_INIT_CONNECT_ERRORS: 0
COUNT_LOCAL_ERRORS: 0
COUNT_UNKNOWN_ERRORS: 0
FIRST_SEEN: 2018-01-31 16:28:19
LAST_SEEN: 2018-01-31 17:02:10
FIRST_ERROR_SEEN: 2018-01-31 16:28:19
LAST_ERROR_SEEN: 2018-01-31 17:02:10
1 row in set (0.00 sec)
ERROR:
No query specified
然后我們用netem與tc 命令在測試服務器上取消網絡延遲模擬，然后去測試連接MySQL數據庫，如下測試所示：
[root@gettestlnx02 ~]# tc qdisc del dev eth0 root netem delay 11000ms
[root@gettestlnx02 ~]# mysql -h10.20.57.24 -utest -p
Enter password:
ERROR 1129 (HY000): Host '192.168.27.180' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'
[root@gettestlnx02 ~]#
此時就能構造出“ERROR 1129 (HY000): Host '192.168.27.180' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'”錯誤了。
解決方案
解決ERROR 1129 (00000): Host 'xxx' is blocked because of many connection errors. Unblock with 'mysqladmin flush-hosts'這個錯誤的方法比較多，不過有些方案都是臨時方案。臨時方案是指標不治本。關鍵還是需要解決網絡錯誤（這個往往需要求助網絡管理人員或系統管理員）
解決方法：
1、將變量max_connection_errors的值設置為一個更大的值
mysql> show variables like 'max_connect_errors';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| max_connect_errors | 3 |
+--------------------+-------+
1 row in set (0.01 sec)
mysql> set global max_connect_errors=150;
Query OK, 0 rows affected (0.00 sec)
mysql>
這個臨時方案只是延遲觸發IP被禁止訪問的條件而已，而且在復雜情況或高並發的情況下，需要設置一個很大的值，否則很容易就會再次被觸發。另外，變量只對當前環境生效，如果重啟就會失效，如果需要永久有效，可以在my.cnf配置文件里面配置。
2：使用flush hosts
mysql> flush hosts;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from performance_schema.host_cache;
Empty set (0.00 sec)
mysql>
當然你也可以mysqladmin flush-hosts 命令清理一下hosts cache信息
[root@DB-Server ~]# mysqladmin --port=3306 -uroot -p flush-host
Enter password:
那么host cache是什么呢？ 官方介紹如下：
The MySQL server maintains a host cache in memory that contains information about clients: IP address, host name, and error information. The server uses this cache for nonlocal TCP connections. It does not use the cache for TCP connections established using a loopback interface address (127.0.0.1 or ::1), or for connections established using a Unix socket file, named pipe, or shared memory.
簡單來說，就是MySQL服務器在內存中維護一個包含客戶端信息的緩存：IP地址，主機名和錯誤信息等。 服務器會將非本地TCP連接信息緩存起來。它不會緩存使用環回接口地址（127.0.0.1或者:: 1）建立的TCP連接，或者使用Unix套接字文件，命名管道或共享內存建立的連接。host cache信息可以通過performance_schema數據庫下的host_cache表查詢。
3：將變量host_cache_size設置為0
其實我想說這是一個最不靠譜的解決方法，只是讓MySQL服務器不記錄host cache信息而已。完全可以忽略這個方法。
mysql> show variables like '%host_cache_size%';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| host_cache_size | 279 |
+-----------------+-------+
1 row in set (0.00 sec)
mysql> set global host_cache_size=0;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from performance_schema.host_cache;
Empty set (0.00 sec)
mysql>
知識點延伸
關於參數max_connect_errors，不要誤解其功能，這個不能作為防止窮舉密碼攻擊的手段。如果您擔心SYN泛濫攻擊，max_connect_errors可能會在特定情況下幫助您。 MySQL 5.6中的PERFORMANCE_SCHEMA改進提供了有關潛在強力攻擊的有意義的信息，但是只有在涉及主機緩存的情況下。關於這個可以參考博文Understanding max_connect_errors
參考資料：
http://mysqlblog.fivefarmers.com/2013/08/08/understanding-max_connect_errors/
https://dev.mysql.com/doc/refman/5.6/en/host-cache.html
https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_max_connect_errors
https://dev.mysql.com/doc/refman/5.7/en/blocked-host.html