Linux常用性能工具功能、用法及原理(一)

本文轉載自查看原文 2019-01-27 22:06 629 計數器/ linux性能工具/ iostat/ sar/ 系統級/ linux/ mpstat/ vmstat/ netstat

Linux性能觀測工具按類別可分為系統級別和進程級別，系統級別對整個系統的性能做統計,而進程級別則具體到進程,為每個進程維護統計信息。

按實現原理分,可分為基於計數器和跟蹤以及剖析。含義如下:

計數器:內核維護的統計數據，通常為無符號整型,用於對發生的事件計數,比如，網絡包接收計數器,磁盤IO計數器等。

跟蹤:跟蹤會收集每一個事件的具體數據,由於跟蹤捕獲事件數據需要消耗CPU且需要較大的存儲空間儲存收集數據,默認不開啟。日志就是一種低頻率的跟蹤，會記錄事件數據。

剖析:對目標采樣或快照來歸納目標特征,如:CPU使用率，通過對程序計數器采樣(一種寄存器，用於指示下一條指令的地址),跟蹤棧找到消耗CPU周期的代碼路徑。剖析也可以通過非計時的硬件事件,如CPU硬件緩存未命中或總線活動，這類信息可以幫助開發人員針對系統資源的使用來優化自己的代碼。

一系統級計數器工具

　　系統級別的工具有vmstat、mpstat、iostat、netstat、sar，這些工具有使用慣例，可選時間間隔和次數。進程級別的工具使用內核為每個進程維護的計數器，有ps、top、pmap。本節主要講基於計數器的系統級性能工具。

1.1 vmstat

　　vmstat是Virtual Meomory Statistics（虛擬內存統計）,用來報告進程、內存、磁盤讀寫、CPU使用整體狀況。常用形式vmstat delay count,其中delay表示采樣間隔,count表示采樣次數，命令執行結果如下圖1.1所示，其中第一行表示系統啟動以來各指標的平均值。

圖1.1 vmstat命令

各個區域的含義如下:

Procs
- r: The number of processes waiting for run time.運行態和就緒態進程數目
- b: The number of processes in uninterruptible sleep.不可中斷進程數目，進程進行系統調用且被阻塞，不可中斷和kill掉時的狀態,絕大多數不可中斷系統調用都比較快完成^[1]，比如mkdir(2)調用過程不會返回EINTR(調用過程被中斷返回),不會被中斷^[2]。
Memory
- swpd: the amount of virtual memory used.使用虛擬內存的大小
- free: the amount of idle memory.空閑內存的大小(物理內存)
- buff: the amount of memory used as buffers。Buff用於緩存磁盤塊數據，如文件系統元數據，文件權限、位置等(metadata)
- cache: the amount of memory used as cache。Cache用於緩存文件內容^[3]
- inact: the amount of inactive memory. (-a option)
- active: the amount of active memory. (-a option)
Swap, 交換區，當內存不夠時,內存中的頁置換到磁盤中^[4]
- si: Amount of memory swapped in from disk (/s).從磁盤交換到內存的大小
- so: Amount of memory swapped to disk (/s).從內存交換到磁盤的大小
IO
- bi: Blocks received from a block device (blocks/s).每秒從塊設備讀出的塊數(如從磁盤讀文件會導致增加)
- bo: Blocks sent to a block device (blocks/s).每秒寫入塊設備的塊數(如寫數據到磁盤會導致增加)
System
- in: The number of interrupts per second, including the clock.每秒中斷次數(包括時鍾中斷)
- cs: The number of context switches per second.每秒上下文切換次數CPU
- us: Time spent running non-kernel code. (user time, including nice time)，用戶態代碼時間,運行計算密集型進程(如壓縮文件)，會導致用戶態CPU增加^[3]。
- sy: Time spent running kernel code. (system time),內核態代碼時間(如頻繁運行系統調用/dev/urandom生成隨機數，會導致sy增加)
- id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.空閑時間，包含了wa時間(cpu空閑的時候時間上會運行空閑線程)。
- wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle.CPU空閑，但有由此CPU發起的IO在進行^[4]
- st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown.Steal time is the percentage of time a virtual CPU waits for a real CPU while the hypervisor is servicing another virtual processor.在雲環境下，物理CPU由多台虛擬機共享,st表示本虛擬CPU等待分配真實CPU的時間，st過高表示可能有其他用戶的虛擬機搶占CPU^[5]。

　　vmstat中的統計值由內核交互文件/proc/meminfo(內存使用信息)、/proc/stat(系統整體進程狀態，如運行態、不可中斷阻塞態進程數,cpu使用情況)、/proc/*/stat(每個進程各自的狀態信息)統計出。

/proc/meminfo文件內容如下,stat文件內容見1.2。

圖1.2 meminfo文件內容

1.2 mpstat

　　mpstat是Multiprocessor Statistics的縮寫,可以查看每個CPU核心的信息,而vmstat只能查看系統整體的CPU信息。

　　用法: mpstat [-P {|ALL}] [internal [count]],-P表示CPU列表,interval表示間隔,count表示采樣次數。執行結果如下圖1.3所示，各字段含義如下:

圖1.3 mpstat結果圖

CPU:Processor number. The keyword all indicates that statistics are calculated as averages among all processors.具體的CPU，all值表示所有CPU的平均值統計信息。
%usr:Show the percentage of CPU utilization that occurred while executing at the user level (application).
%nice:Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.nice值發生變動(即優先級發生變化)的進程的CPU時間,nice值為負表示進程的優先級變高。
%sys:Show the percentage of CPU utilization that occurred while executing at the system level (kernel). Note that this does not include time spent servicing hardware and software interrupts.內核代碼運行時間,不包括中斷服務時間。
%iowait:Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.含義見1.1.1節vmstat的wa字段。
%irq:Show the percentage of time spent by the CPU or CPUs to service hardware interrupts.CPU花在硬中斷服務時間
%soft:Show the percentage of time spent by the CPU or CPUs to service software interrupts. CPU花在軟中斷服務時間
%steal:Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.含義見1.1.1節vmstat的st字段
%guest:Show the percentage of time spent by the CPU or CPUs to run a virtual processor.CPU運行虛擬處理器所花費的時間
%gnice:Show the percentage of time spent by the CPU or CPUs to run a niced guest.
%idle:Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.CPU空閑時間，和vmstat有點不同，這里不包含iowait時間，但vmstat包含。

　　vmstat數據來源/proc/stat,此文件包含所有cpu的活動,是開機累積到現在的統計值。文件內容如下圖1.4所示,cpu表示總的信息,cpu0、cpu1…表示各cpu的信息，從前到后的值含義分別為user、nice、system、idle、iowait、irq、softirq、stealstolen、guest,以時間片為單位,intr后的第一個值表示系統自啟動到現在的中斷次數，后面的每一個值表示某種中斷類型發生的次數,ctxt表示cpu上下文切換的次數,btime表示系統啟動到現在的時間(實際觀察前后兩次未發生變化), processes表示系統啟動到現在以來創建的任務個數, procs_running表示當前運行隊列任務個數, procs_blocked表示阻塞任務個數，softirq表示總軟中斷次數及各種類型軟中斷的次數^[6]。

圖1.4 /proc/stat文件內容

1.3 iostat　

　　iostat是I/O statistics（輸入/輸出統計）的縮寫，用來動態監視系統的磁盤操作活動。

　　用法: iostat [ -p [ device [,...] | ALL ] ] [ device [...] | ALL ] [ interval [ count ] ], -p表示設備列表,interval表示間隔,count表示采樣次數。執行結果如下圖1.5和1.6所示，其中選項-d表示顯示設備統計(Display the device utilization report),-x表示顯示詳細信息(Display extended statistics)。各字段含義如下:

圖1.5 iostat設備io統計信息

圖1.6 iostat設備io詳細統計信息

Device:This column gives the device (or partition) name as listed in the /dev directory.設備名稱
Tps:Indicate the number of transfers per second that were issued to the device. A transfer is an I/O request to the device. Multiple logical requests can be combined into a single I/O request to the device. A transfer is of indeterminate size.每秒I/O次數(即每秒設備請求次數)
Blk_read/s (kB_read/s, MB_read/s),Indicate the amount of data read from the device expressed in a number of blocks (kilobytes, megabytes) per second. Blocks are equivalent to sectors and therefore have a size of 512 bytes.每秒讀數據多少
Blk_wrtn/s (kB_wrtn/s, MB_wrtn/s),Indicate the amount of data written to the device expressed in a number of blocks (kilobytes, megabytes) per second.每秒寫數據多少
Blk_read (kB_read, MB_read),The total number of blocks (kilobytes, megabytes) read.interval間隔內讀數據總量
Blk_wrtn (kB_wrtn, MB_wrtn),The total number of blocks (kilobytes, megabytes) written. interval間隔內寫數據總量
rrqm/s,The number of read requests merged per second that were queued to the device.每秒合並讀操作的數目,當兩個讀操作讀相鄰的數據塊時，會被合並為一個請求，提高效率, 合並的操作通常是I/O scheduler（也叫elevator）負責的。(順序讀和隨機讀對比,順序讀此值會比較大)^[7]
wrqm/s,The number of write requests merged per second that were queued to the device. .每秒合並寫操作的數目,當兩個寫操作寫相鄰的數據塊時，會被合並為一個請求，提高效率(順序寫和隨機寫對比,順序寫此值會比較大)^[7]
r/s,The number (after merges) of read requests completed per second for the device.每秒讀操作次數
w/s,The number (after merges) of write requests completed per second for the device.每秒寫操作次數
rsec/s (rkB/s, rMB/s),The number of sectors (kilobytes, megabytes) read from the device per second.每秒讀數據量大小，可以以sector和kB,MB為單位
wsec/s (wkB/s, wMB/s),The number of sectors (kilobytes, megabytes) written to the device per second. 每秒寫數據量大小，可以以sector和kB,MB為單位
avgrq-sz,The average size (in sectors) of the requests that were issued to the device. 每個IO的平均扇區數，即所有請求的平均大小，以扇區（512字節）為單位
avgqu-sz,The average queue length of the requests that were issued to the device.平均請求隊列長度
await,The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.平均每個io的時間(包括了在等列中等待的時間和磁盤處理的時間)^[8]。如果I/O模式很隨機(磁盤服務時間)、I/O負載(隊列中等待時間)比較高，會導致磁頭亂跑，尋道時間長，那么相應地await要估算得大一些；如果I/O模式是順序讀寫，只有單一進程產生I/O負載，那么尋道時間和旋轉延遲都可以忽略不計，主要考慮傳輸時間，相應地await就應該很小，甚至不到1毫秒, 對磁盤陣列來說，因為有硬件緩存，寫操作不等落盤就算完成，所以寫操作的service time大大加快了，如果磁盤陣列的寫操作不在一兩個毫秒以內就算慢的了；讀操作則未必，不在緩存中的數據仍然需要讀取物理硬盤，單個小數據塊的讀取速度跟單盤差不多^[9]。
r_await,The average time (in milliseconds) for read requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them. 平均每個讀io的時間(包括了在等列中等待的時間和磁盤處理的時間)
w_await,The average time (in milliseconds) for write requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them. 平均每個寫io的時間(包括了在等列中等待的時間和磁盤處理的時間)
svctm,The average service time (in milliseconds) for I/O requests that were issued to the device. Warning! Do not trust this field any more. This field will be removed in a future sysstat version.(已被廢棄)
%util,Percentage of elapsed time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.工作時間或者繁忙時間占總時間的百分比,表示該設備有I/O（即非空閑）的時間比率，不考慮I/O有多少，只考慮有沒有.現代硬盤都有並行處理的功能，當%util為100%時並不能說明磁盤處理已飽和，達到瓶頸,因為可能可以接收更多的並行處理請求,iostat沒有衡量磁盤是否飽和的指標^[9]。

　　iostat的統計數據主要來源與文件/proc/diskstats,文件內容如下圖1.7所示:

圖1.7 diskstats文件內容

　　每行各列的含義依序如下表1.1所示,各值為系統啟動到現在為止的累加值:

表1.1 diskstats含義表

Field	Value	Quoted	解釋
F1	253	major number	主設備號
F2	0	minor mumber	從設備號
F3	vda	device name	設備名
F4(rd_ios)	1012759	reads completed	讀完成次數
F5(rd_merges)	3418	reads merged	讀合並次數
F6(rd_sectors)	48466695	sectors read	讀取的扇區數
F7(rd_ticks)	4027016	milliseconds spent reading	所有讀操作時間的累加值(ms單位)
F8(wr_ios)	19073830	writes completed	寫操作次數
F9(writes merged)	33764041	writes merged	寫合並次數
F10(wr_sectors)	2195716912	sectors written	寫入的扇區數
F11(wr_ticks)	120769824	milliseconds spent writing	所有寫操作消耗的時間累加(ms單位)
F12(in_flight)	0	I/Os currently in progress	未完成io的數目
F13(io_ticks)	10982660	milliseconds spent doing I/Os	該設備用於處理I/O的自然時間(wall-clock time)
F14(time_in_queue)	124778072	weighted # of milliseconds spent doing I/Os	對字段F13(io_ticks)的加權值

　　in_flight表示系統未完成io的任務數，當io進入請求隊列時加1(I/O請求進入隊列時，而不是提交給硬盤設備時),當io任務完成時減1, in_flight包括在隊列中等待和正在進行io的任務。

　　rd_ticks和wr_ticks是把每一個IO消耗時間累加起來，但是硬盤設備一般可以並行處理多個IO，因此，rd_ticks和wr_ticks之和一般會比自然時間（wall-clock time）要大。而io_ticks 不關心隊列中有多少個IO在排隊，它只關心設備有IO的時間。即不考慮IO有多少，只考慮IO有沒有。在實際運算中，in_flight不是0的時候保持計時，而in_flight 等於0的時候，時間不累加到io_ticks。

　　其中,io_ticks這個字段被iostat用來計算%util，而time_in_queue這個字段被iostat用來計算avgqu-sz，即平均隊列長度。^[9]

1.4 netstat

　　netstat主要用來查看網絡連接、路由表、網卡統計信息。

1.4.1 查看網絡連接(等價ss)

　　其中-t,-u等表示網絡活動的協議,-l表示socket處於listening狀態,-a表示顯示所有狀態(包括listening狀態和非listening狀態),-n表示主機、端口、用戶用數字表示，不用解析成名稱,-e表示詳細信息,-p顯示socket的進程pid和名稱,-c會每秒連續輸出網絡信息。執行結果如下圖1.8所示:

圖1.8 netstat查看網絡連接

　　部分字段含義如下:

Recv-Q.
- Established: The count of bytes not copied by the user program connected to this socket. 指收到的數據還在緩存中，還沒被進程讀取，這個值就是還沒被進程讀取的 byte數。
- Listening: Since Kernel 2.6.18 this column contains the current syn backlog. syn backlog表示處於半連接狀態的隊列，接收到客戶端的syn請求，則會進入此隊列。
Send-Q.
- Established: The count of bytes not acknowledged by the remote host. 發送隊列中沒有被遠程主機確認的 bytes 數。
- Listening: Since Kernel 2.6.18 this column contains the maximum size of the syn backlog.半連接隊列的最大大小
State.The state of the socket. Since there are no states in raw mode and usually no states used in UDP and UDPLite, this column may be left blank. Normally this can be one of several values:
- ESTABLISHED.The socket has an established connection.三次握手，第三次握手發送ack后的狀態
- SYN_SENT.The socket is actively attempting to establish a connection.發送syn后的狀態
- SYN_RECV.A connection request has been received from the network.接收到syn+ack后的狀態
- FIN_WAIT1.The socket is closed, and the connection is shutting down.socket被關閉，正在關閉連接(四次揮手,第一次揮手,主動發送FIN后的狀態)
- FIN_WAIT2.Connection is closed, and the socket is waiting for a shutdown from the remote end.連接關閉,等待遠端關閉信號(四次揮手,第二次揮手,接收到FIN的響應ACK后的狀態)
- TIME_WAIT.The socket is waiting after close to handle packets still in the network. FIN_WAIT2狀態接收到遠端的FIN包,變為TIME_WAIT狀態(四次揮手,收到第三次揮手FIN信號)
- CLOSE .The socket is not being used.
- CLOSE_WAIT.The remote end has shut down, waiting for the socket to close.對端已關閉,但本端向遠端仍可以發送數據。(四次揮手，第二次揮手,向對端發送FIN信號的響應)
- LAST_ACK.The remote end has shut down, and the socket is closed. Waiting for acknowledgement. CLOSE_WAIT狀態下，向遠端發送FIN信號(四次揮手,第三次揮手,向對端發送FIN)
- LISTEN.The socket is listening for incoming connections. Such sockets are not included in the output unless you specify the --listening (-l) or --all (-a) option.
- CLOSING.Both sockets are shut down but we still don't have all our data sent.
- UNKNOWN.The state of the socket is unknown.

　　Recv-Q和Send-Q一般情況下為0值，可接收短暫情況情況下為非0情況,假如長時間為非0,表明隊列有堆積,假如時Recv-Q為非0,應用程序接收不急，可能是拒絕服務攻擊,假如是Send-Q為非0,可能是應用程序發送進程和服務端接收進程速度不匹配，發送太快或接收太慢^[13]。

　　netstat的信息來源於/proc/net文件夾,關於tcp連接的信息來源於/proc/net/tcp文件,內容如下圖1.9所示:

圖1.9 /proc/net/tcp文件

1.4.2 查看路由表信息(等價ip route)

　　用法:netstat {--route|-r} [address_family_options] [--extend|-e[--extend|-e]] [--verbose|-v] [--numeric|-n] [--numeric-hosts] [--numeric-ports] [--numeric-users] [--continuous|-c] [delay]

　　執行結果如下圖1.10所示，路由表信息來源於/proc/net/route文件:

圖1.10 內核路由表

1.4.3 查看網卡統計信息(等價ip -s link)

　　用法:netstat {--interfaces|-I|-i} [--all|-a] [--extend|-e] [--verbose|-v] [--program|-p] [--numeric|-n] [--numeric-hosts] [--numeric-ports] [--numeric-users] [--continuous|-c] [delay]

　　執行結果如下圖1.11:

圖1.11 網卡活動統計

1.5 sar

　　sar(System Activity Reporter)用於統計和收集系統活動，包括CPU、內存、網絡、IO等多方面，是一個比較全面的工具。sar收集系統數據保存在/var/log/sa/sadd(dd為日期)文件。

1.5.1 查看CPU使用情況

　　懷疑CPU存在瓶頸，可用 sar -u和 sar -q 等來查看。sar -u和mpstat相似，字段含義參考1.1.2。sar -p用於查看隊列長度和負載信息，執行結果如下圖1.12：

圖1.12 sar -q執行結果

runq-sz.Run queue length (number of tasks waiting for run time).運行態和就緒態任務數
plist-sz.Number of tasks in the task list.處於任務隊列的任務總數
ldavg-1.System load average for the last minute. The load aver‐age is calculated as the average number of runnable or running tasks (R state), and the number of tasks in unin‐terruptible sleep (D state) over the specified interval.過去一分鍾負載(處於運行態和就緒態及不可中斷狀態任務數)
ldavg-5.System load average for the past 5 minutes.過去五分鍾負載
ldavg-15.System load average for the past 15 minutes.過去十五分鍾負載
Blocked.Number of tasks currently blocked, waiting for I/O to complete.等待io完成的處於阻塞態任務數。

1.5.2 查看內存使用情況

　　懷疑內存存在瓶頸，可用sar -B、sar -r 和 sar -W 等來查看，Sar -B,統計頁交換情況。執行結果如圖1.13:

圖1.13 sar -B執行結果

pgpgin/s，Total number of kilobytes the system paged in from disk per second.每秒從磁盤讀入頁的大小。
pgpgout/s，Total number of kilobytes the system paged out to disk per second.每秒寫入磁盤的頁大小。
fault/s，Number of page faults (major + minor) made by the system per second. This is not a count of page faults that generate I/O, because some page faults can be resolved without I/O.包括major和minor,major缺頁需要進行磁盤io，但minor不需要,minor falut頁在內存中，只是還未指定到進程地址空間，比如多進程代碼共享，此時不需要重新從磁盤讀入代碼。
majflt/s，Number of major faults the system has made per second,those which have required loading a memory page from disk. 每秒產生的要進行磁盤IO的缺頁次數^[16]
pgfree/s，Number of pages placed on the free list by the system per second. 每秒放置在可用列表中的頁數。
pgscank/s，Number of pages scanned by the kswapd daemon per second. 表示的是Linux當中的負責內存回收的線程kswapd掃描的頁的數量,
pgscand/s，Number of pages scanned directly per second. 每秒直接掃描的頁數
pgsteal/s，Number of pages the system has reclaimed from cache (pagecache and swapcache) per second to satisfy its memory demands. 系統滿足自身的內存需要，每秒從緩存回收的頁數.
%vmeff，Calculated as pgsteal / pgscan, this is a metric of the efficiency of page reclaim. If it is near 100% then almost every page coming off the tail of the inactive list is being reaped. If it gets too low (e.g. less than 30%) then the virtual memory is having some difficulty. This field is displayed as zero if no pages have been scanned during the interval of time. pgscan= pgscank/s+ pgscand/s，衡量分頁回收有效性的指標，理想情況應該是100%或0(未發生頁面掃描時), 表示被掃描的頁當中, 有多少頁的數據被踢出去了換成其他頁數據了^[17]

　　page fault和swap不同，當要訪問的代碼或數據沒有虛擬地址和物理地址對應，會產生page fault；當內存空間不足，需要釋放部分內存空間加載其它數據時，會將內存中的部分頁swap到磁盤。^[18]

　　sar -r統計內存使用情況。執行結果如下圖1.14所示:

圖1.14 sar -r執行結果

kbcommit.Amount of memory in kilobytes needed for current workload. This is an estimate of how much RAM/swap is needed to guarantee that there never is out of memory.已經申請的內存的大小(可能未分配)，對應/proc/meminfo中的Committed_AS
%commit.Percentage of memory needed for current workload in relation to the total amount of memory (RAM+swap). This number may be greater than 100% because the kernel usually overcommits memory. Kbcommit和RAM+swap的百分比，可能大於100%,內核允許申請的內存大小大於總內存大小。^[19]
kbactive.Amount of active memory in kilobytes (memory that has been used more recently and usually not reclaimed unless absolutely necessary).活躍內存大小
kbinact.Amount of inactive memory in kilobytes (memory which has been less recently used. It is more eligible to be reclaimed for other purposes).不活躍內存大小
kbdirty.Amount of memory in kilobytes waiting to get written back to the disk. 等待寫回硬盤的內存數量

　　sar -W統計swap in和out信息，執行結果如下圖1.15所示:

圖1.15 sar -W執行結果

pswpin/s,Total number of swap pages the system brought in per second.每秒swap in數目
pswpout/s,Total number of swap pages the system brought out per second.每秒swap out數目

1.5.3 查看設備IO情況

　　懷疑I/O存在瓶頸，可用 sar -b和 sar -d 等來查看,sar -b統計IO和傳輸速率信息，執行結果如下圖1.16所示:

圖1.16 sar -b執行結果

tps.Total number of transfers per second that were issued to physical devices. A transfer is an I/O request to a physical device. Multiple logical requests can be combined into a single I/O request to the device. A transfer is of indeterminate size.每秒IO請求數
rtps.Total number of read requests per second issued to physical devices.都請求數
wtps.Total number of write requests per second issued to physical devices.寫請求數
bread/s.Total amount of data read from the devices in blocks per second. Blocks are equivalent to sectors with 2.4 kernels and newer and therefore have a size of 512 bytes. With older kernels, a block is of indeterminate size.每秒讀取數據塊個數
bwrtn/s.Total amount of data written to devices in blocks per second.每秒寫入磁盤的數據塊數.

　　sar -d統計磁盤塊活動,執行結果如下圖1.17所示，和iostat字段含義類似。

圖1.17 sar -d執行結果

1.5.4 查看網絡流量情況

　　sar –n用於統計網絡接收包信息，用法如下sar -n { keyword [,...] | ALL },關鍵有DEV, EDEV, FC, ICMP, EICMP, ICMP6, EICMP6, IP, EIP, IP6, EIP6, NFS, NFSD, SOCK, SOCK6, SOFT, TCP, ETCP, UDP and UDP6，分別從不同的角度展示網絡統計信息,這里主要說明下DEV關鍵字,sar –n DEV執行結果如下圖1.18所示:

圖1.18 DEV結果

IFACE.Name of the network interface for which statistics are reported.網絡接口名
rxpck/s.Total number of packets received per second.每秒收包數
txpck/s.Total number of packets transmitted per second.每秒發包數
rxkB/s.Total number of kilobytes received per second.每秒收到的kB數
txkB/s.Total number of kilobytes transmitted per second.每秒發送的kB數
rxcmp/s.Number of compressed packets received per second (for cslip etc.).每秒收到的壓縮數據包
txcmp/s. Number of compressed packets transmitted per second.每秒發送的壓縮數據包
rxmcst/s. Number of multicast packets received per second.每秒收到的廣播數據包數

　　sar性能數據來源於/var/log/sa/saDD、/var/log/sa/saYYYYMMDD、以及/proc 和 /sys中的文件。