中秋的時候正在外面愉快的在外賣喝着咖啡玩電腦。。。。。。突發 redis 報警從 sentry 應用端曝出的錯誤
MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled,
because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option).
Please check the Redis logs for details about the RDB error.
於是又開始愉快的處理問題了，看上去像是執行 rdb 快照持久化的時候出現的問題，上到 redis 機器查看日志定位詳細問題
420:M 14 Sep 15:56:27.067 # Can't save in background: fork: Cannot allocate memory 420:M 14 Sep 15:56:33.071 * 10000 changes in 60 seconds. Saving... 420:M 14 Sep 15:56:33.072 # Can't save in background: fork: Cannot allocate memory 420:M 14 Sep 15:56:39.079 * 10000 changes in 60 seconds. Saving... 420:M 14 Sep 15:56:39.080 # Can't save in background: fork: Cannot allocate memory 420:M 14 Sep 15:56:45.083 * 10000 changes in 60 seconds. Saving... 420:M 14 Sep 15:56:45.083 # Can't save in background: fork: Cannot allocate memory 420:M 14 Sep 15:56:51.094 * 10000 changes in 60 seconds. Saving... 420:M 14 Sep 15:56:51.095 # Can't save in background: fork: Cannot allocate memory 420:M 14 Sep 15:56:57.002 * 10000 changes in 60 seconds. Saving...
可以很明顯的發現應該是嘗試 fork 的時候內存不夠，並沒有被 linux 內核放行。
這里有兩個點我認為需要注意一下，一個是 redis 在默認配置的情況是下是開啟參數
stop-writes-on-bgsave-error yes
也就是 如果 bgsave 存儲快照失敗，那么 redis 將阻止數據繼續寫入，如果將這個設置成 False 那么即使是 bgsave 快照寫入磁盤失敗，也不會讓 redis 立即對外停止服務。
但是無法 bgsave 讓數據落盤始終是隱患，要是機器一重啟，就完蛋了。所以我嘗試查詢一些熱修復的手段來修復這個問題。
最終 linux 端有一個參數 vm.overcommit_memory 可以解決這個問題默認參數是 0 ，它有三個值可以配置。
這時候就是內存不足，到了這里，操作系統要怎么辦，就要祭出我們的主角“overcommit_memory”參數了（/proc/sys/vm/overcommit_memory）； vm.overcommit_memory = 0 啟發策略 比較 此次請求分配的虛擬內存大小和系統當前空閑的物理內存加上swap，決定是否放行。系統在為應用進程分配虛擬地址空間時，會判斷當前申請的虛擬地址空間大小是否超過剩余內存大小，如果超過，則虛擬地址空間分配失敗。因此，也就是如果進程本身占用的虛擬地址空間比較大或者剩余內存比較小時，fork、malloc等調用可能會失敗。 vm.overcommit_memory = 1 允許overcommit 直接放行，系統在為應用進程分配虛擬地址空間時，完全不進行限制，這種情況下，避免了fork可能產生的失敗，但由於malloc是先分配虛擬地址空間，而后通過異常陷入內核分配真正的物理內存，在內存不足的情況下，這相當於完全屏蔽了應用進程對系統內存狀態的感知，即malloc總是能成功，一旦內存不足，會引起系統OOM殺進程，應用程序對於這種后果是無法預測的。 vm.overcommit_memory = 2 禁止overcommit 根據系統內存狀態確定了虛擬地址空間的上限，由於很多情況下，進程的虛擬地址空間占用遠大於其實際占用的物理內存，這樣一旦內存使用量上去以后，對於一些動態產生的進程(需要復制父進程地址空間)則很容易創建失敗，如果業務過程沒有過多的這種動態申請內存或者創建子進程，則影響不大，否則會產生比較大的影響 。這種情況下系統所能分配的內存不會超過上面提到的CommitLimit大小，如果這么多資源已經用光，那么后面任何嘗試申請內存的行為都會返回錯誤，這通常意味着此時沒法運行任何新程序。 ———————————————— 版權聲明：本文為CSDN博主「朱清震」的原創文章，遵循 CC 4.0 BY-SA 版權協議，轉載請附上原文出處鏈接及本聲明。 原文鏈接：https://blog.csdn.net/zqz_zqz/article/details/53384854
所以這里 bgsave 我們 redis 應用會嘗試對主進程進行 fork ，然后內存不夠申請未被內核放行。所以 hotfix 我嘗試將參數 vm.overcommit_memory 設置成 1 直接進行放行。
/etc/sysctl.conf vm.overcommit_memory=1 sysctl -p
生效，再看日志發現就可以成功了。
這里我找到官方 FAQ 也對類似問題有描述
Background saving fails with a fork() error under Linux even if I have a lot of free RAM!
Short answer:
echo 1 > /proc/sys/vm/overcommit_memory:)And now the long one:
Redis background saving schema relies on the copy-on-write semantic of fork in modern operating systems: Redis forks (creates a child process) that is an exact copy of the parent. The child process dumps the DB on disk and finally exits. In theory the child should use as much memory as the parent being a copy, but actually thanks to the copy-on-write semantic implemented by most modern operating systems the parent and child process will share the common memory pages. A page will be duplicated only when it changes in the child or in the parent. Since in theory all the pages may change while the child process is saving, Linux can't tell in advance how much memory the child will take, so if the
overcommit_memorysetting is set to zero fork will fail unless there is as much free RAM as required to really duplicate all the parent memory pages, with the result that if you have a Redis dataset of 3 GB and just 2 GB of free memory it will fail.Setting
overcommit_memoryto 1 tells Linux to relax and perform the fork in a more optimistic allocation fashion, and this is indeed what you want for Redis.A good source to understand how Linux Virtual Memory works and other alternatives for
overcommit_memoryandovercommit_ratiois this classic from Red Hat Magazine, "Understanding Virtual Memory". Beware, this article had1and2configuration values forovercommit_memoryreversed: refer to the proc(5) man page for the right meaning of the available values.
后來 hotfix 之后，我們清理了一些很久未能釋放的大 key，將內存恢復到比較小的水平。就很穩了，這次問題發生之后沒有無腦進行重啟，而是迅速通過一定的思路來查詢問題，感覺自己解決問題的方法稍微成熟了一點點。
Reference:
https://zhuanlan.zhihu.com/p/36872365 fork 的原理及實現
https://stackoverflow.com/questions/11752544/redis-bgsave-failed-because-fork-cannot-allocate-memory redis bgsave failed because fork Cannot allocate memory
https://www.freebsd.org/doc/zh_CN/books/handbook/configtuning-sysctl.html 12.11. 用 sysctl 進行調整
https://blog.csdn.net/zqz_zqz/article/details/53384854 redis Can’t save in background: fork: Cannot allocate memory 解決及原理
https://redis.io/topics/faq 官方 FAQ