Centos XFS文件系統分配內存失敗 kernel: XFS: xfs_fsr() possible memory allocation deadlock size 65552 in kmem_realloc (mode:0x250)

根據文章能找到臨時解決方案 《XFS:kmem_alloc中可能的內存分配死鎖》

相關命令：

echo 1 > /proc/sys/vm/drop_caches   

 

順便了解到可以部署 xfs_gurad 做后台守護進程解決，過程如下：

找到了基於ubuntu 的 ansible_role文件 ，鏈接：https://github.com/Rheinwerk/ansible-role-xfs_guard

由於不適合 Centos 決定做微調，流程如下：

1. 創建需要的文件

/usr/local/sbin/xfs-guard

#!/bin/bash
# vim: ft=sh ts=4 noet

function emit_bosun {
  if [[ -x /usr/local/bin/emit_bosun ]]; then
    /usr/local/bin/emit_bosun ${*}
  fi
}

if [[ -z "${SEARCHTERM}" || -z "${MONITORED_LOG}" || -z "${STATEFILE}" || -z "${THRESHOLD_SECONDS}" ]]; then
        echo "${0}: Missing one or more required environment variables." >&2
        exit 1;
fi

tail -n 1 -F "${MONITORED_LOG}" | grep --line-buffered "${SEARCHTERM}" |  awk '{ print systime(); fflush(); }' | while read -r found_at_time; do
        emit_bosun -m xfs-guard.events.xfs-allocation-deadlock.found  -v 1 --tags service=centerdevice -r gauge -u None -d "Found a kernel log line regarding XFS memory allocation drop."
        if [[ -r "${STATEFILE}" ]]; then
                last_dropped_caches_at=$(cat "${STATEFILE}")
        else
                last_dropped_caches_at=0
        fi
        difference=$(( ${found_at_time} - ${last_dropped_caches_at}))

        if [[ ${difference} -ge ${THRESHOLD_SECONDS}  ]]; then
                echo 2 > /proc/sys/vm/drop_caches
                date +%s > "${STATEFILE}"
                logger -t xfs-guard "Dropped slab cache"
                emit_bosun -m xfs-guard.events.drop.slab -v 1 -r gauge -u None -d "1 = dropped slab cache -- echo 2 > /proc/sys/vm/drop_caches"
        else
                logger -t xfs-guard "Withstood the urge to drop slab cache, because last time was less than ${THRESHOLD_SECONDS}s ago."
                emit_bosun -m xfs-guard.events.skipped.slab -v 1 -r gauge -u None -d "1 = didn't drop slab cache, because requested too often in quick succession"
        fi
done

 

/etc/sysconfig/xfs-guard

SEARCHTERM='kernel: XFS.*possible memory allocation deadlock size.*in kmem_realloc'
MONITORED_LOG="/var/log/messages"
STATEFILE="/run/xfs-guard.txt"
THRESHOLD_SECONDS=10
ENABLED=1

 

創建系統服務

/lib/systemd/system/xfs-guard.service 

[Unit]
Description=Monitor and mitigate XFS memory allocation freezes 
DefaultDependencies=no

[Service]
Type=simple
ExecStart=/usr/local/sbin/xfs-guard
EnvironmentFile=-/etc/sysconfig/xfs-guard

 

刷新服務並增加開機啟動

systemctl daemon-reload   

systemctl enable xfs-guard.service