Centos7中systemd-journald占用內存過高問題

 

systemd-journald進程占用內存過高

同事反饋生產環境DB proxy代理層服務器間隔性地出現內存使用率過高告警，每次都通過重啟部分進程處理，但過不了多久，問題會再次復現。

 

 

 

1.查看消耗memory的top進程

首先，在問題出現期間通過top命令按M可查看占用內存較高的top進程數，發現下面的systemd-journald進程消耗內存非常高，14G的機型單獨該進程就占用了7.6G內存：

 

[root@dbhost ~]# top
top - 10:12:13 up 170 days, 22:27,  2 users,  load average: 2.22, 2.87, 3.41
Tasks: 423 total,   1 running, 390 sleeping,   1 stopped,  31 zombie
%Cpu(s): 11.3 us,  9.0 sy,  0.0 ni, 75.3 id,  0.2 wa,  0.0 hi,  3.7 si,  0.4 st
KiB Mem : 14027920 total,   547544 free, 11278808 used,  2201568 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  2307820 avail Mem


  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                      
 8466 root      20   0 7957192   7.6g   5624 S   0.6 56.5 406:22.26 systemd-journal    <<<<< 占用7.6G                                                                                                                    
 1380 tdsql     20   0 3811756 645664   3328 S  44.9  4.6  32506:23 mysql-proxy                                                                                                                                  
30062 user_00   20   0  443428 248500    800 S   0.3  1.8 423:53.62 l5_agent                                                                                                                                     
 2438 tdsql     20   0 1042784  92972   3528 S   0.3  0.7   3330:18 mysql-proxy                                                                                                                                  
85161 tdsql     20   0  985404  86112   2272 S   0.6  0.6   3911:48 mysql-proxy                                                                                                                                  
54000 tdsql     20   0  862524  82176   2576 S   9.6  0.6  15644:05 mysql-proxy


[root@dbhost ~]# ps -ef | grep systemd-journald | grep -v grep
root      8466     1  0 Feb18 ?        06:46:28 usr/lib/systemd/systemd-journald
[root@dbhost ~]#

 

 

2.查看進程說明

通過man我們可以直接查看CentOS想法服務進程的介紹，systemd-journald的man信息大概就是說該服務進程用於收集並存儲系統的登錄信息。

[root@dbhost ~]# man systemd-journald
Description
systemd-journald is a system service that collects and stores logging data. 
It creates and maintains structured, indexed journals based on logging information that is received from a variety of sources:


Kernel log messages, via kmsg


Simple system log messages, via the libc syslog(3) call


Structured system log messages via the native Journal API, see sd_journal_print(3)


Standard output and standard error of service units. For further details see below.


Audit records, originating from the kernel audit subsystem

1.查看進程配置

既然是服務進程，我們不妨可以先看看改進程的配置文件，看看能否找到突破口，如下：

 

[root@dbhost ~]# cat etc/systemd/journald.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.


[Journal]
#Storage=auto    <<<該配置未開啟
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
RuntimeMaxUse=256M   <<<並不起作用
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K
[root@dbhost ~]#

 

 

2.查看配置說明

上面的配置信息中，可以看到該進程收集到的信息存儲參數並沒有配置，使用的是默認值auto，而默認值會/run/log/journal文件不存在的時候，會將信息保存在內存當中，。

[root@dbhost ~]# man 5 journald.conf
OPTIONS
       All options are configured in the "[Journal]" section:


       Storage=
           Controls where to store journal data. One of "volatile", "persistent", "auto" and "none". If "volatile", journal log data will be stored only in memory, i.e. below the run/log/journal
           hierarchy (which is created if needed). If "persistent", data will be stored preferably on disk, i.e. below the var/log/journal hierarchy (which is created if needed), with a fallback to
           run/log/journal (which is created if needed), during early boot and if the disk is not writable.  "auto" is similar to "persistent" but the directory var/log/journal is not created if needed,
           so that its existence controls where log data goes.  "none" turns off all storage, all log data received will be dropped. Forwarding to other targets, such as the console, the kernel log
           buffer, or a syslog socket will still work however. Defaults to "auto".

 

查看journal文件不存在

[root@dbhost ~]# ls -l var/log/journal
ls: cannot access var/log/journal: No such file or directory
[root@dbhost ~]#

 

 

四、解決方法及總結

把配置項Storage參數值改為none然后重啟進程即可

1）把配置項Storage參數值改為none
[root@dbhost ~]# grep -i storage /etc/systemd/journald.conf
Storage=none
[root@dbhost ~]# 


2）重啟進程生效
systemctl restart systemd-journald

 

 

總結

由於db代理層的服務器用戶登錄登出頻繁，此時如果systemd-journald進程使用默認值，會將大量的用戶登錄信息存儲到主機內存中，從而消耗大量的主機內存，如果日常運維管理對相關信息沒要求，建議可以在CentOS 7版本中systemd-journal在主機初始化中優化掉。