一、問題引入
最近一台mysql服務器報錯：ERROR 1129 (00000): Host 'xxx' is blocked because of many connection errors. Unblock with 'mysqladmin flush-hosts'。
本質原因：
因為同一個IP在短時間內產生太多中斷的數據庫連接（超過max_connect_errors的最大值）而導致的
If more than this many successive connection requests from a host are interrupted without a successful connection, the server blocks that host from further connections. You can unblock blocked hosts by flushing the host cache. To do so, issue a FLUSH HOSTS statement or execute a mysqladmin flush-hosts command. If a connection is established successfully within fewer than max_connect_errors attempts after a previous connection was interrupted, the error count for the host is cleared to zero. However, once a host is blocked, flushing the host cache is the only way to unblock it. The default is 100.
翻譯：如果MySQL服務器連續接收到了來自於同一個主機的請求，而且這些連續的請求全部都沒有成功的建立連接就被中斷了，當這些連續的請求的累計值大於max_connect_errors的設定值時，MySQL服務器就會阻止這台主機后續的所有請求。相信一開始你看到這些資料，也會被“many successive connection requests from a host are interrupted without a successful connection”給弄懵，其實這個就是因為由於網絡異常而中止數據庫連接
There seems to be confusion around that variable. It does not really block hosts for repeated invalid passwords but for aborted connections due to network errors.
二、實驗驗證
#創建測試用戶 mysql> grant all privileges on *.* to test@'%' identified by 'mysql'; Query OK, 0 rows affected, 1 warning (0.01 sec) #查看變量max_connect_errors並設置值 mysql> show variables like '%max_connect_errors%'; +--------------------+-------+ | Variable_name | Value | +--------------------+-------+ | max_connect_errors | 100 | +--------------------+-------+ 1 row in set (0.13 sec) mysql> set global max_connect_errors=3; Query OK, 0 rows affected (0.00 sec) mysql> show variables like '%max_connect_errors%'; +--------------------+-------+ | Variable_name | Value | +--------------------+-------+ | max_connect_errors | 3 | +--------------------+-------+ 1 row in set (0.00 sec) #在另外一台測試機器，以錯誤的密碼去連接這個MySQL數據庫，如下所示，即使前面輸入了三次錯誤密碼，第四次輸入是也沒有碰到上面錯誤。那么可以排除這個變量與密碼錯誤輸入有關系 [root@docker02 ~]# mysql -utest -pmysql11 -h 192.168.11.10 ERROR 1045 (28000): Access denied for user 'test'@'192.168.11.11' (using password: YES) [root@docker02 ~]# mysql -utest -pmysql11 -h 192.168.11.10 ERROR 1045 (28000): Access denied for user 'test'@'192.168.11.11' (using password: YES) [root@docker02 ~]# mysql -utest -pmysql11 -h 192.168.11.10 ERROR 1045 (28000): Access denied for user 'test'@'192.168.11.11' (using password: YES) [root@docker02 ~]# mysql -utest -pmysql11 -h 192.168.11.10 ERROR 1045 (28000): Access denied for user 'test'@'192.168.11.11' (using password: YES) [root@docker02 ~]# mysql -utest -pmysql11 -h 192.168.11.10 ERROR 1045 (28000): Access denied for user 'test'@'192.168.11.11' (using password: YES)
關於某個IP輸入了錯誤密碼，MySQL會在performance_schema數據庫下的host_cache表中記錄。它會累計記錄在COUNT_AUTHENTICATION_ERRORS字段
host_cache的字段是統計被視為“阻塞”的連接錯誤的數量（根據max_connect_errors系統變量進行評估）。 只計算協議握手錯誤，並且僅用於通過驗證的主機（HOST_VALIDATED = YES）
SUM_CONNECT_ERRORS：The number of connection errors that are deemed “blocking” (assessed against the max_connect_errors system variable). Only protocol handshake errors are counted, and only for hosts that passed validation (HOST_VALIDATED = YES).
MySQL客戶端與數據庫建立連接需要發起三次握手協議，正常情況下，這個時間非常短，但是一旦網絡異常，網絡超時等因素出現，就會導致這個握手協議無法完成，MySQL有個參數connect_timeout，它是MySQL服務端進程mysqld等待連接建立完成的時間，單位為秒。如果超過connect_timeout時間范圍內，仍然無法完成協議握手話，MySQL客戶端會收到異常，異常消息類似於: Lost connection to MySQL server at 'XXX', system error: errno，該變量默認是10秒
mysql> show variables like 'connect_timeout'; +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | connect_timeout | 10 | +-----------------+-------+ 1 row in set (0.01 sec)
構造一個網絡超時引起的數據庫連接被中斷案例吧，我們用Linux下的netem與tc命令模擬構造出復雜環境下的網絡傳輸延時案例，如下設置后，此時從測試服務器去訪問MySQL服務器，都會出現延時11秒
[root@docker02 ~]# ping 192.168.11.10 PING 192.168.11.10 (192.168.11.10) 56(84) bytes of data. 64 bytes from 192.168.11.10: icmp_seq=1 ttl=64 time=0.686 ms 64 bytes from 192.168.11.10: icmp_seq=2 ttl=64 time=0.472 ms 64 bytes from 192.168.11.10: icmp_seq=3 ttl=64 time=0.465 ms 64 bytes from 192.168.11.10: icmp_seq=4 ttl=64 time=0.512 ms ^C --- 192.168.11.10 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3002ms rtt min/avg/max/mdev = 0.465/0.533/0.686/0.094 ms [root@docker02 ~]# tc qdisc add dev eth0 root netem delay 11000ms [root@docker02 ~]# [root@docker02 ~]# ping 192.168.11.10 PING 192.168.11.10 (192.168.11.10) 56(84) bytes of data. 64 bytes from 192.168.11.10: icmp_seq=1 ttl=64 time=11001 ms 64 bytes from 192.168.11.10: icmp_seq=2 ttl=64 time=11000 ms From 192.168.11.11 icmp_seq=10 Destination Host Unreachable From 192.168.11.11 icmp_seq=11 Destination Host Unreachable From 192.168.11.11 icmp_seq=12 Destination Host Unreachable From 192.168.11.11 icmp_seq=13 Destination Host Unreachable 64 bytes from 192.168.11.10: icmp_seq=3 ttl=64 time=11001 ms 64 bytes from 192.168.11.10: icmp_seq=4 ttl=64 time=11000 ms 64 bytes from 192.168.11.10: icmp_seq=5 ttl=64 time=11000 ms 64 bytes from 192.168.11.10: icmp_seq=6 ttl=64 time=11000 ms 64 bytes from 192.168.11.10: icmp_seq=7 ttl=64 time=11000 ms
在測試服務器連接MySQL數據庫，如下所示（注意，如果你是在通過ssh連接這台服務器的話，此時在測試服務器上操作會相當慢。當然你也可以在MySQL服務器模擬網絡延時，或者你將connect_timeout和網絡延時都設小一點）
[root@docker02 ~]# mysql -utest -pmysql -h 192.168.11.10 ERROR 2013 (HY000): Lost connection to MySQL server at 'reading authorization packet', system error: 0
由於網絡延時超過10秒，導致連接MySQL失敗，此時，你在MySQL服務器上查詢host_cache表時,那么你就會看到SUM_CONNECT_ERRORS變成1了，COUNT_HANDSHAKE_ERRORS也變成了1
反復這樣折騰三次，那么你會看到SUM_CONNECT_ERRORS變成3了，COUNT_HANDSHAKE_ERRORS也變成了3了
用netem與tc 命令在測試服務器上取消網絡延遲模擬，然后去測試連接MySQL數據庫
此時就能構造出“ERROR 1129 (HY000): Host 'xxx' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'”錯誤了。
三、解決方案
解決方法比較多，不過有些方案都是臨時方案。臨時方案是指標不治本。關鍵還是需要解決網絡錯誤（這個往往需要求助網絡管理人員或系統管理員）
1）將變量max_connection_errors的值設置為一個更大的值
這個臨時方案只是延遲觸發IP被禁止訪問的條件而已，而且在復雜情況或高並發的情況下，需要設置一個很大的值，否則很容易就會再次被觸發。另外，變量只對當前環境生效，如果重啟就會失效，如果需要永久有效，可以在my.cnf配置文件里面配置。
mysql> show variables like 'max_connect_errors'; +--------------------+-------+ | Variable_name | Value | +--------------------+-------+ | max_connect_errors | 3 | +--------------------+-------+ 1 row in set (0.03 sec) mysql> set global max_connect_errors=150; Query OK, 0 rows affected (0.00 sec)
2）使用flush hosts
#方式一 mysql> flush hosts; Query OK, 0 rows affected (0.00 sec) mysql> select * from performance_schema.host_cache\G Empty set (0.00 sec) #方式二 [root@docker01 ~]# mysqladmin -uroot -pmysql flush-host
host cache解釋說明：
The MySQL server maintains a host cache in memory that contains information about clients: IP address, host name, and error information. The server uses this cache for nonlocal TCP connections. It does not use the cache for TCP connections established using a loopback interface address (127.0.0.1 or ::1), or for connections established using a Unix socket file, named pipe, or shared memory.
MySQL服務器在內存中維護一個包含客戶端信息的緩存：IP地址，主機名和錯誤信息等。 服務器會將非本地TCP連接信息緩存起來。它不會緩存使用環回接口地址（127.0.0.1或者:: 1）建立的TCP連接，或者使用Unix套接字文件，命名管道或共享內存建立的連接。host cache信息可以通過performance_schema數據庫下的host_cache表查詢。
3）將變量host_cache_size設置為0
最不靠譜的解決方法，只是讓MySQL服務器不記錄host cache信息而已。完全可以忽略這個方法
mysql> show variables like '%host_cache_size%'; +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | host_cache_size | 279 | +-----------------+-------+ 1 row in set (0.00 sec) mysql> set global host_cache_size=0; Query OK, 0 rows affected (0.00 sec) mysql> show variables like '%host_cache_size%'; +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | host_cache_size | 0 | +-----------------+-------+ 1 row in set (0.00 sec) mysql> select * from performance_schema.host_cache; Empty set (0.00 sec)
四、參考資料
http://mysqlblog.fivefarmers.com/2013/08/08/understanding-max_connect_errors/
https://dev.mysql.com/doc/refman/5.6/en/host-cache.html
https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_max_connect_errors