Linux 運行進程實時監控pidstat命令詳解

簡介

 pidstat主要用於監控全部或指定進程占用系統資源的情況，如CPU，內存、設備IO、任務切換、線程等。pidstat首次運行時顯示自系統啟動開始的各項統計信息，之后運行pidstat將顯示自上次運行該命令以后的統計信息。用戶可以通過指定統計的次數和時間來獲得所需的統計信息。

 

實例講解

默認參數

執行pidstat，將輸出系統啟動后所有活動進程的cpu統計信息：

 

Linux 2.6.32.12-0.7-default (linux)             06/18/12        _x86_64_

11:37:19          PID    %usr %system  %guest    %CPU   CPU  Command
……
11:37:19        11452    0.00    0.00    0.00    0.00     2  bash
11:37:19        11509    0.00    0.00    0.00    0.00     3  dd

pidstat命令指定采樣周期和采樣次數，命令形式為”pidstat [option] interval [count]”，以下pidstat輸出以2秒為采樣周期，輸出10次cpu使用統計信息：

pidstat 2 10

 

 

cpu使用情況統計(-u)

使用-u選項，pidstat將顯示各活動進程的cpu使用統計，執行”pidstat -u”與單獨執行”pidstat”的效果一樣。

 

內存使用情況統計(-r)

使用-r選項，pidstat將顯示各活動進程的內存使用統計：

Linux 2.6.32.12-0.7-default (linux)             06/18/12        _x86_64_

15:08:18          PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
15:08:19        13084 133835.00      0.00 15720284 15716896  96.26  mmmm
15:08:20        13084  35807.00      0.00 15863504 15849756  97.07  mmmm
15:08:21        13084  19273.87      0.00 15949040 15792944  96.72  mmmm

minflt/s: 每秒次缺頁錯誤次數(minor page faults)，次缺頁錯誤次數意即虛擬內存地址映射成物理內存地址產生的page fault次數
majflt/s: 每秒主缺頁錯誤次數(major page faults)，當虛擬內存地址映射成物理內存地址時，相應的page在swap中，這樣的page fault為major page fault，一般在內存使用緊張時產生
VSZ:      該進程使用的虛擬內存(以kB為單位)
RSS:      該進程使用的物理內存(以kB為單位)
%MEM:     該進程使用內存的百分比
Command:  拉起進程對應的命令

 

IO情況統計(-d)

使用-d選項，我們可以查看進程IO的統計信息：

Linux 2.6.32.12-0.7-default (linux)             06/18/12        _x86_64_

17:11:36          PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
17:11:37        14579 124988.24      0.00      0.00  dd

17:11:37          PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
17:11:38        14579 105441.58      0.00      0.00  dd

kB_rd/s: 每秒進程從磁盤讀取的數據量(以kB為單位)
kB_wr/s: 每秒進程向磁盤寫的數據量(以kB為單位)
Command: 拉起進程對應的命令

 

針對特定進程統計(-p)

使用-p選項，我們可以查看特定進程的系統資源使用情況：

Linux 2.6.32.12-0.7-default (linux)             06/18/12        _x86_64_

18:26:17          PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
18:26:18            1      0.00      0.00   10380    640   0.00  init
18:26:19            1      0.00      0.00   10380    640   0.00  init
……

 

pidstat常用命令

使用pidstat進行問題定位時，以下命令常被用到：

pidstat -u 1

pidstat -r 1

pidstat -d 1

以上命令以1秒為信息采集周期，分別獲取cpu、內存和磁盤IO的統計信息。

 

 

pidstat實際上也是將/proc/pid下的統計信息統籌后展現給用戶。

pidstat例子

1. 列出IO統計信息：-d

2. 列出內存使用統計，page fault：-r

3. CPU統計信息：-u

4. 上下文切換統計信息：-w

5. 指定輸出的維度

指定命令： -C command

指定進程號：-p { pid [,...] | SELF | ALL }

將所有列輸出到單行、便於導入單張表中：-h

按每個CPU核的統計：-I

按所有CPU核統計：默認

列出命令的完整內容，包括參數：-l

列出線程統計信息：-t

按進程維度單獨統計、按進程全局+子進程單獨統計、按進程全局、單獨統計同時按子任務單獨統計：-T { TASK | CHILD | ALL }

實例，統計5秒的值並輸出

#pidstat -d -r -u -w -l -h -p ALL 5 1|less # Time PID %usr %system %guest %CPU CPU minflt/s majflt/s VSZ RSS %MEM kB_rd/s kB_wr/s kB_ccwr/s cswch/s nvcswch/s Command 1488541252 1 0.00 0.00 0.00 0.00 0 0.00 0.00 19348 1512 0.00 0.00 0.00 0.00 0.00 0.00 /sbin/init 1488541252 2 0.00 0.00 0.00 0.00 50 0.00 0.00 0 0 0.00 0.00 0.00 0.00 0.00 0.00 kthreadd 1488541252 3 0.00 0.00 0.00 0.00 0 0.00 0.00 0 0 0.00 0.00 0.00 0.00 0.59 0.00 migration/0 1488541252 4 0.00 0.00 0.00 0.00 0 0.00 0.00 0 0 0.00 0.00 0.00 0.00 0.00 0.00 ksoftirqd/0 1488541252 5 0.00 0.00 0.00 0.00 0 0.00 0.00 0 0 0.00 0.00 0.00 0.00 0.00 0.00 migration/0 1488541252 6 0.00 0.00 0.00 0.00 0 0.00 0.00 0 0 0.00 0.00 0.00 0.00 0.00 0.00 watchdog/0 ...... 1488541264 18523 0.00 0.20 0.00 0.20 5 10.78 0.00 137774940 3002232 0.57 0.00 0.00 0.00 1.96 0.20 /home/digoal/pgsql9.6/bin/postgres 1488541264 18549 0.00 0.00 0.00 0.00 61 0.00 0.00 140892 2028 0.00 0.00 0.00 0.00 0.00 0.00 postgres: logger process 1488541264 18552 0.00 0.00 0.00 0.00 0 0.00 0.00 137779116 5509520 1.04 0.00 0.00 0.00 0.00 0.00 postgres: checkpointer process 1488541264 18554 0.00 0.00 0.00 0.00 20 0.00 0.00 137776328 1058572 0.20 0.00 0.00 0.00 3.92 0.00 postgres: writer process 1488541264 18556 0.00 0.00 0.00 0.00 15 0.00 0.00 137774940 19120 0.00 0.00 0.00 0.00 3.92 0.00 postgres: wal writer process 1488541264 18557 0.00 0.00 0.00 0.00 21 2.94 0.00 137779092 3404 0.00 0.00 0.00 0.00 3.92 0.00 postgres: autovacuum launcher process 1488541264 18559 0.00 0.00 0.00 0.00 11 0.00 0.00 142988 2092 0.00 0.00 0.00 0.00 0.00 0.00 postgres: archiver process last was 0000000100000030000000B2 1488541264 18561 0.00 0.00 0.00 0.00 53 4.90 0.00 143556 2624 0.00 0.00 317.65 317.65 4.12 0.20 postgres: stats collector process 

 

 

 

NAME
pidstat - Report statistics for Linux tasks.

SYNOPSIS
pidstat [ -d ] [ -h ] [ -I ] [ -l ] [ -r ] [ -s ] [ -t ] [ -U [ username ] ] [ -u ] [ -V ] [ -w ] [ -C comm ] [ -p { pid [,...] | SELF | ALL } ] [ -T { TASK | CHILD | ALL }
] [ interval [ count ] ]

DESCRIPTION
The pidstat command is used for monitoring individual tasks currently being managed by the Linux kernel. It writes to standard output activities for every task selected
with option -p or for every task managed by the Linux kernel if option -p ALL has been used. Not selecting any tasks is equivalent to specifying -p ALL but only active
tasks (tasks with non-zero statistics values) will appear in the report.

The pidstat command can also be used for monitoring the child processes of selected tasks. Read about option -T below.

The interval parameter specifies the amount of time in seconds between each report. A value of 0 (or no parameters at all) indicates that tasks statistics are to be
reported for the time since system startup (boot). The count parameter can be specified in conjunction with the interval parameter if this one is not set to zero. The
value of count determines the number of reports generated at interval seconds apart. If the interval parameter is specified without the count parameter, the pidstat command
generates reports continuously.

You can select information about specific task activities using flags. Not specifying any flags selects only CPU activity.

OPTIONS
-C comm
Display only tasks whose command name includes the string comm. This string can be a regular expression.

-d Report I/O statistics (kernels 2.6.20 and later only). The following values may be displayed:

UID
The real user identification number of the task being monitored.

USER
The name of the real user owning the task being monitored.

PID
The identification number of the task being monitored.

kB_rd/s
Number of kilobytes the task has caused to be read from disk per second.

kB_wr/s
Number of kilobytes the task has caused, or shall cause to be written to disk per second.

kB_ccwr/s
Number of kilobytes whose writing to disk has been cancelled by the task. This may occur when the task truncates some dirty pagecache. In this case, some IO
which another task has been accounted for will not be happening.

Command
The command name of the task.

-h Display all activities horizontally on a single line. This is intended to make it easier to be parsed by other programs.

-I In an SMP environment, indicate that tasks CPU usage (as displayed by option -u ) should be divided by the total number of processors.

-l Display the process command name and all its arguments.

-p { pid [,...] | SELF | ALL }
Select tasks (processes) for which statistics are to be reported. pid is the process identification number. The SELF keyword indicates that statistics are to be
reported for the pidstat process itself, whereas the ALL keyword indicates that statistics are to be reported for all the tasks managed by the system.

-r Report page faults and memory utilization.

When reporting statistics for individual tasks, the following values may be displayed:

UID
The real user identification number of the task being monitored.

USER
The name of the real user owning the task being monitored.

PID
The identification number of the task being monitored.

minflt/s
Total number of minor faults the task has made per second, those which have not required loading a memory page from disk.

majflt/s
Total number of major faults the task has made per second, those which have required loading a memory page from disk.

VSZ
Virtual Size: The virtual memory usage of entire task in kilobytes.

RSS
Resident Set Size: The non-swapped physical memory used by the task in kilobytes.

%MEM
The tasks's currently used share of available physical memory.

Command
The command name of the task.

When reporting global statistics for tasks and all their children, the following values may be displayed:

UID
The real user identification number of the task which is being monitored together with its children.

USER
The name of the real user owning the task which is being monitored together with its children.

PID
The identification number of the task which is being monitored together with its children.

minflt-nr
Total number of minor faults made by the task and all its children, and collected during the interval of time.

majflt-nr
Total number of major faults made by the task and all its children, and collected during the interval of time.

Command
The command name of the task which is being monitored together with its children.

-s Report stack utilization. The following values may be displayed:

UID
The real user identification number of the task being monitored.

USER
The name of the real user owning the task being monitored.

PID
The identification number of the task being monitored.

StkSize
The amount of memory in kilobytes reserved for the task as stack, but not necessarily used.

StkRef
The amount of memory in kilobytes used as stack, referenced by the task.

Command
The command name of the task.

-T { TASK | CHILD | ALL }
This option specifies what has to be monitored by the pidstat command. The TASK keyword indicates that statistics are to be reported for individual tasks (this is
the default option) whereas the CHILD keyword indicates that statistics are to be globally reported for the selected tasks and all their children. The ALL keyword
indicates that statistics are to be reported for individual tasks and globally for the selected tasks and their children.

Note: Global statistics for tasks and all their children are not available for all options of pidstat. Also these statistics are not necessarily relevant to current
time interval: The statistics of a child process are collected only when it finishes or it is killed.

-t Also display statistics for threads associated with selected tasks.

This option adds the following values to the reports:

TGID
The identification number of the thread group leader.

TID
The identification number of the thread being monitored.

-U [ username ]
Display the real user name of the tasks being monitored instead of the UID. If username is specified, then only tasks belonging to the specified user are displayed.

-u Report CPU utilization.

When reporting statistics for individual tasks, the following values may be displayed:

UID
The real user identification number of the task being monitored.

USER
The name of the real user owning the task being monitored.

PID
The identification number of the task being monitored.

%usr
Percentage of CPU used by the task while executing at the user level (application), with or without nice priority. Note that this field does NOT include time
spent running a virtual processor.

%system
Percentage of CPU used by the task while executing at the system level (kernel).

%guest
Percentage of CPU spent by the task in virtual machine (running a virtual processor).

%CPU
Total percentage of CPU time used by the task. In an SMP environment, the task's CPU usage will be divided by the total number of CPU's if option -I has been
entered on the command line.

CPU
Processor number to which the task is attached.

Command
The command name of the task.

When reporting global statistics for tasks and all their children, the following values may be displayed:

UID
The real user identification number of the task which is being monitored together with its children.

USER
The name of the real user owning the task which is being monitored together with its children.

PID
The identification number of the task which is being monitored together with its children.

usr-ms
Total number of milliseconds spent by the task and all its children while executing at the user level (application), with or without nice priority, and col‐
lected during the interval of time. Note that this field does NOT include time spent running a virtual processor.

system-ms
Total number of milliseconds spent by the task and all its children while executing at the system level (kernel), and collected during the interval of time.

guest-ms
Total number of milliseconds spent by the task and all its children in virtual machine (running a virtual processor).

Command
The command name of the task which is being monitored together with its children.

-V Print version number then exit.

-w Report task switching activity (kernels 2.6.23 and later only). The following values may be displayed:

UID
The real user identification number of the task being monitored.

USER
The name of the real user owning the task being monitored.

PID
The identification number of the task being monitored.

cswch/s
Total number of voluntary context switches the task made per second. A voluntary context switch occurs when a task blocks because it requires a resource that
is unavailable.

nvcswch/s
Total number of non voluntary context switches the task made per second. A involuntary context switch takes place when a task executes for the duration of
its time slice and then is forced to relinquish the processor.

Command
The command name of the task.

ENVIRONMENT
The pidstat command takes into account the following environment variable:

S_TIME_FORMAT
If this variable exists and its value is ISO then the current locale will be ignored when printing the date in the report header. The pidstat command will use the
ISO 8601 format (YYYY-MM-DD) instead.

EXAMPLES
pidstat 2 5
Display five reports of CPU statistics for every active task in the system at two second intervals.

pidstat -r -p 1643 2 5
Display five reports of page faults and memory statistics for PID 1643 at two second intervals.

pidstat -C "fox|bird" -r -p ALL
Display global page faults and memory statistics for all the processes whose command name includes the string "fox" or "bird".

pidstat -T CHILD -r 2 5
Display five reports of page faults statistics at two second intervals for the child processes of all tasks in the system. Only child processes with non-zero statis‐
tics values are displayed.

BUGS
/proc filesystem must be mounted for the pidstat command to work.

FILES
/proc contains various files with system statistics.

AUTHOR
Sebastien Godard (sysstat <at> orange.fr)

SEE ALSO
sar(1), top(1), ps(1), mpstat(1), iostat(1), vmstat(8)

http://pagesperso-orange.fr/sebastien.godard/

 

Linux MARCH 2013 PIDSTAT(1)