MySQL運行內存不足時應采取的措施？

本文轉載自查看原文 2019-01-24 00:27 1127 MySQL DBA

排除故障指南:MySQL運行內存不足時應采取的措施？

天一閣@ 老葉茶館 1周前

導讀

排除故障指南：MySQL運行內存不足時應采取的措施？

翻譯團隊：知數堂藏經閣項目 - 天一閣

團隊成員：天一閣-冷鋒、天一閣-Judy 、天一閣-神諭

譯文校稿：葉師傅

原文出處：《What To Do When MySQL Runs Out of Memory: Troubleshooting Guide》

https://www.percona.com/blog/2018/06/28/what-to-do-when-mysql-runs-out-of-memory-troubleshooting-guide/

原文作者：Alexander Rubin

關鍵詞：memory、memory leaks、Memory Usage、MySQL server memory usage、MySQL Troubleshooting、Troubleshooting MySQL、troubleshooting tips

Troubleshooting crashes is never a fun task, especially if MySQL does not report the cause of the crash. For example, when MySQL runs out of memory. Peter Zaitsev wrote a blog post in 2012: Troubleshooting MySQL Memory Usage with a lots of useful tips. With the new versions of MySQL (5.7+) and performance_schema we have the ability to troubleshoot MySQL memory allocation much more easily。
崩潰故障診斷絕不是一個有趣的任務，尤其如果MySQL沒有報告崩潰的原因時，例如MySQL運行時內存溢出。Peter Zaitsev 在2012年寫了一篇文章Troubleshooting MySQL Memory Usage 里面有很多有用的提示，使用新版本MySQL(5.7+)結合performance_schema，我們可以更輕松地解決MySQL內存分配問題。

In this blog post I will show you how to use it.

First of all, there are 3 major cases when MySQL will crash due to running out of memory:

MySQL tries to allocate more memory than available because we specifically told it to do so. For example: you did not set innodb_buffer_pool_size correctly. This is very easy to fix
There is some other process(es) on the server that allocates RAM. It can be the application (java, python, php), web server or even the backup (i.e. mysqldump). When the source of the problem is identified, it is straightforward to fix.
Memory leaks in MySQL. This is a worst case scenario, and we need to troubleshoot.

在這篇博文中，我將向你展示如何使用它。

首先，MySQL因為內存溢出發生崩潰主要有以下三種情況：

MySQL試圖分配比可用內存更多的內存，因為我們特意告訴它這樣做；比如你沒有正確的設置innodb_buffer_pool_size，那這種情況很好解決。
服務器上有其他一些進程分配了RAM內存，可能是應用程序(java/python/php)、web服務器，或者甚至備份(比如mysqldump)，確定問題的根源后，可以直接修復。
MySQL內存泄漏，這是最糟糕的情況，這時需要我們進行故障診斷。

Where to start troubleshooting MySQL memory leaks

從哪里開始診斷MySQL內存泄漏的問題

Here is what we can start with (assuming it is a Linux server):

假設是一個linux服務器，我們可以從以下開始:

Part 1: Linux OS and config check

Identify the crash by checking mysql error log and Linux log file (i.e. /var/log/messages or /var/log/syslog). You may see an entry saying that OOM Killer killed MySQL. Whenever MySQL has been killed by OOM “dmesg” also shows details about the circumstances surrounding it.
Check the available RAM:

free -g
cat /proc/meminfo

Check what applications are using RAM: “top” or “htop” (see the resident vs virtual memory)
Check mysql configuration: check /etc/my.cnf or in general /etc/my* (including /etc/mysql/* and other files). MySQL may be running with the different my.cnf (run ps ax| grep mysql )
Run vmstat 5 5 to see if the system is reading/writing via virtual memory and if it is swapping
For non-production environments we can use other tools (like Valgrind, gdb, etc) to examine MySQL usage

第一部分: Linux 系統和配置檢查

通過檢查mysql error日志和linux日志(比如,/var/log/messages 或者 /var/log/syslog)確認崩潰。你可能會看到一條條目說OOM Killer殺死了MySQL，每當MySQL被OOM殺死時，“dmesg”也會顯示有關它周圍情況的詳細信息。
檢查可用的RAM內存:

free -g
cat /proc/meminfo

檢查什么程序在使用內存:"top"或者htop(看resident和virtual列)
檢查mysql的配置:檢查/etc/my.cnf或者一般的/etc/my*(包括/etc/mysql/*和其他文件)；
MySQL 可能跟着不同的my.cnf運行（用ps ax | grep mysql)
運行vmstat 5 5 查看系統是否通過虛擬內存進行讀寫以及是否正在進行swap交換
對於非生產環境，我們可以使用其他工具(如Valgrind、gdb等)來檢查MySQL的使用情況。

Part 2: Checks inside MySQL
Now we can check things inside MySQL to look for potential MySQL memory leaks.
MySQL allocates memory in tons of places. Especially:

Table cache
Performance_schema (run: show engine performance_schema status and look at the last line). That may be the cause for the systems with small amount of RAM, i.e. 1G or less
InnoDB (run show engine innodb status and check the buffer pool section, memory allocated for buffer_pool and related caches)
Temporary tables in RAM (find all in-memory tables by running: select * from information_schema.tables where engine='MEMORY' )
Prepared statements, when it is not deallocated (check the number of prepared commands via deallocate command by running show global status like ‘ Com_prepare_sql';show global status like 'Com_dealloc_sql' )

第二部分: 檢查MySQL內部

現在我們可以檢查MySQL內部的東西來尋找潛在的MySQL內存泄漏情況:
MySQL在很多地方分配內存，尤其：

表緩存
Performance_schema(運行:show engine performance_schema status 然后看最后一行)，這可能在系統RAM比較少(1G或更少)時的可能原因。
InnoDB(運行show engine innodb status 檢查 buffer pool部分,為buffer pool及相關緩存分配的內存)
內存中的臨時表(查看所有內存表:select * from information_schema.tables where engine='MEMORY')
預處理語句，當他們沒有被釋放時(通過運行show global status like 'Com_prepare_sql'和show global status like 'Com_dealloc_sql'來檢查通過deallocate命令釋放的預處理語句)

The good news is: starting with MySQL 5.7 we have memory allocation in performance_schema. Here is how we can use it.
好消息是，從5.7開始我們可以通過performance_schema查看內存的分配情況。下面就展示如何使用它：

First, we need to enable collecting memory metrics. Run:

UPDATE setup_instruments SET ENABLED = 'YES' WHERE NAME LIKE 'memory/%';

Run the report from sys schema:

select event_name, current_alloc, high_alloc from sys.memory_global_by_current_bytes where current_count > 0;

Usually this will give you the place in code when memory is allocated. It is usually self-explanatory. In some cases we can search for bugs or we might need to check the MySQL source code.
首先，我們需要啟用收集內存指標，運行如下語句：

UPDATE setup_instruments SET ENABLED = 'YES' WHERE NAME LIKE 'memory/%';

運行sys schema里面的報告

select event_name,current_alloc,high_alloc from sys.memory_global_by_current_bytes where current_count > 0;

通常，這將在分配內存時為你提供代碼，它通常是不言自明的。在某些情況下，我們可以搜索錯誤，或者我們可能需要檢查MySQL源代碼。

For example, for the bug where memory was over-allocated in triggers (https://bugs.mysql.com/bug.php?id=86821) the select shows:

例如，有一個過度為觸發器分配內存的bug：

https://bugs.mysql.com/bug.php?id=86821

查詢的顯示如下：

The largest chunk of RAM is usually the buffer pool but ~3G in stored procedures seems to be too high.
分配最大一塊內存通常是buffer pool，但是約3G的存儲過程似乎有點太高了.

According to the MySQL source code documentation, sp_head represents one instance of a stored program which might be of any type (stored procedure, function, trigger, event). In the above case we have a potential memory leak.
根據MySQL source code documentation,sp_head表示存儲程序里面的一個實例(比如存儲過程、函數、觸發器，及事件)。在上面的例子，我們有潛在的內存泄漏的風險。

In addition we can get a total report for each higher level event if we want to see from the birds eye what is eating memory:
另外，我們想要鳥瞰什么吃掉了內存，我們可以獲得每個事件更高級別活動的總體報告。