TCP的連接狀態對於我們web服務器來說是至關重要的，尤其是並發量ESTAB；或者是syn_recv值，假如這個值比較大的話我們可以認為是不是受到了攻擊，或是是time_wait值比較高的話，我們要考慮看我們內核是否需要調優，太高的time_wait值的話會占用太多端口，要是端口少的話后果不堪設想
一、TCP狀態介紹
1、man netstat查看TCP的各種狀態信息描述：
LISTEN -偵聽來自遠方TCP端口的連接請求； SYN-SENT -在發送連接請求后等待匹配的連接請求； SYN-RECEIVED -在收到和發送一個連接請求后等待對連接請求的確認； ESTABLISHED -代表一個打開的連接，數據可以傳送給用戶； FIN-WAIT-1 -等待遠程TCP的連接中斷請求，或先前的連接中斷請求的確認； FIN-WAIT-2 -從遠程TCP等待連接中斷請求； CLOSE-WAIT -等待從本地用戶發來的連接中斷請求； CLOSING -等待遠程TCP對連接中斷的確認； LAST-ACK -等待原來發向遠程TCP的連接中斷請求的確認； TIME-WAIT -等待足夠的時間以確保遠程TCP接收到連接中斷請求的確認； CLOSED -沒有任何連接狀態；
2、監控原理
[root@Node1 ~]# /bin/netstat -an|awk '/^tcp/{++S[$NF]}END{for(a in S) print a,S[a]}' //通過netstat獲取相關值 LISTEN 10 ESTABLISHED 1 TIME_WAIT 178 [root@Node1 ~]#
二、監控腳本編寫
1、編寫腳本，放於 /etc/zabbix/zabbix_agentd.d/目錄下
1 [root@Node1 zabbix_agentd.d]# cat tcp_status.sh 2 #!/bin/bash 3 #This script is used to get tcp and udp connetion status 4 #tcp status 5 metric=$1 6 tmp_file=/tmp/tcp_status.txt 7 /bin/netstat -an|awk '/^tcp/{++S[$NF]}END{for(a in S) print a,S[a]}' > $tmp_file 8 case $metric in 9 closed) 10 output=$(awk '/CLOSED/{print $2}' $tmp_file) 11 if [ "$output" == "" ];then 12 echo 0 13 else 14 echo $output 15 fi 16 ;; 17 listen) 18 output=$(awk '/LISTEN/{print $2}' $tmp_file) 19 if [ "$output" == "" ];then 20 echo 0 21 else 22 echo $output 23 fi 24 ;; 25 synrecv) 26 output=$(awk '/SYN_RECV/{print $2}' $tmp_file) 27 if [ "$output" == "" ];then 28 echo 0 29 else 30 echo $output 31 fi 32 ;; 33 synsent) 34 output=$(awk '/SYN_SENT/{print $2}' $tmp_file) 35 if [ "$output" == "" ];then 36 echo 0 37 else 38 echo $output 39 fi 40 ;; 41 established) 42 output=$(awk '/ESTABLISHED/{print $2}' $tmp_file) 43 if [ "$output" == "" ];then 44 echo 0 45 else 46 echo $output 47 fi 48 ;; 49 timewait) 50 output=$(awk '/TIME_WAIT/{print $2}' $tmp_file) 51 if [ "$output" == "" ];then 52 echo 0 53 else 54 echo $output 55 fi 56 ;; 57 closing) 58 output=$(awk '/CLOSING/{print $2}' $tmp_file) 59 if [ "$output" == "" ];then 60 echo 0 61 else 62 echo $output 63 fi 64 ;; 65 closewait) 66 output=$(awk '/CLOSE_WAIT/{print $2}' $tmp_file) 67 if [ "$output" == "" ];then 68 echo 0 69 else 70 echo $output 71 fi 72 ;; 73 lastack) 74 output=$(awk '/LAST_ACK/{print $2}' $tmp_file) 75 if [ "$output" == "" ];then 76 echo 0 77 else 78 echo $output 79 fi 80 ;; 81 finwait1) 82 output=$(awk '/FIN_WAIT1/{print $2}' $tmp_file) 83 if [ "$output" == "" ];then 84 echo 0 85 else 86 echo $output 87 fi 88 ;; 89 finwait2) 90 output=$(awk '/FIN_WAIT2/{print $2}' $tmp_file) 91 if [ "$output" == "" ];then 92 echo 0 93 else 94 echo $output 95 fi 96 ;; 97 *) 98 echo -e "\e[033mUsage: sh $0 [closed|closing|closewait|synrecv|synsent|finwait1|finwait2|listen|established|lastack|timewait]\e[0m" 99 esac
賦予腳本執行權限：
[root@Node1 ~]# chmod a+x /etc/zabbix/zabbix_agentd.d/tcp_status.sh //賦予執行權限 [root@Node1 ~]# ll /etc/zabbix/zabbix_agentd.d/tcp_status.sh -rwxr-xr-x 1 root root 2502 Jan 18 09:48 /etc/zabbix/zabbix_agentd.d/tcp_status.sh [root@Node1 ~]#
2、添加zabbix配置文件，放於 /etc/zabbix/zabbix_agentd.d/目錄下（agent的配置文件 /etc/zabbix/zabbix_agentd.conf 中定義了其他key的包含目錄）創建配置文件tcp_status.conf
[root@Node1 ~]# cat /etc/zabbix/zabbix_agentd.d/tcp_status.conf UserParameter=tcp.status[*],/etc/zabbix/zabbix_agentd.d/tcp_status.sh "$1" //腳本路徑 [root@Node1 ~]#
3、確保配置Agent配置文件開啟自定義參數UnsafeUserParameters=1
[root@Node1 ~]# grep -n "^[a-Z]" /etc/zabbix/zabbix_agentd.conf 13:PidFile=/var/run/zabbix/zabbix_agentd.pid 32:LogFile=/var/log/zabbix/zabbix_agentd.log 43:LogFileSize=0 57:DebugLevel=3 97:Server=172.17.21.208 138:ServerActive=172.17.21.208 149:Hostname=Node1.contoso.com 267:Include=/etc/zabbix/zabbix_agentd.d/*.conf 286:UnsafeUserParameters=1 //1代表允許，0代表關閉
4、重啟zabbix-agent服務
[root@Node1 ~]# systemctl restart zabbix-agent.service
備注：因為腳本是把tcp的一些信息存放在/tmp/下，為了zabbix可以讀取到我們設置zabbix可以讀的權限，確保屬主與屬組都為zabbix即可
[root@Node1 ~]# chown zabbix.zabbix /tmp/tcp_status.txt //改變屬主與屬主 [root@Node1 ~]# ll /tmp/tcp_status.txt -rw-rw-r-- 1 zabbix zabbix 38 Jan 18 11:32 /tmp/tcp_status.txt
5、在zabbix servere服務器上測試,是否能正常獲取數據
[root@Node3 ~]# zabbix_get -s 172.17.21.206 -p 10050 -k "tcp.status[listen]" 0 [root@Node3 ~]# zabbix_get -s 172.17.21.206 -p 10050 -k "tcp.status[timewait]" 183 [root@Node3 ~]# zabbix_get -s 172.17.21.206 -p 10050 -k "tcp.status[established]" 2 [root@Node3 ~]#
三、WEB界面配置
1、導入事先准備好的TCP Connection Status模板，Configuration-->Templates-->Import，選擇相應的模板，點擊Import即可
2、關聯模板，Configuration-->Hosts-->Node1--Templates，選擇模板,點擊Updtae即可
3、查看TCP狀態：
Monitoring-->Graphs-->Host(Node1)-->Grahp(TCP Status)
附模板：Template_TCP_Status.xml
<?xml version="1.0" encoding="UTF-8"?> <zabbix_export> <version>3.4</version> <date>2018-01-18T11:10:22Z</date> <groups> <group> <name>Templates</name> </group> </groups> <templates> <template> <template>Template TCP Connection Status</template> <name>Template TCP Connection Status</name> <description/> <groups> <group> <name>Templates</name> </group> </groups> <applications> <application> <name>TCP Status</name> </application> </applications> <items> <item> <name>CLOSED</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[closed]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>CLOSE_WAIT</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[closewait]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>CLOSING</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[closing]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>ESTABLISHED</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[established]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>FIN_WAIT1</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[finwait1]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>FIN_WAIT2</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[finwait2]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>LAST_ACK</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[lastack]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>LISTEN</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[listen]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>SYN_RECV</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[synrecv]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>SYN_SENT</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[synsent]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>TIME_WAIT</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[timewait]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> </items> <discovery_rules/> <httptests/> <macros/> <templates/> <screens/> </template> </templates> <triggers> <trigger> <expression>{Template TCP Connection Status:tcp.status[listen].last()}>500</expression> <recovery_mode>0</recovery_mode> <recovery_expression/> <name>There are too many TCP LISTEN status</name> <correlation_mode>0</correlation_mode> <correlation_tag/> <url/> <status>0</status> <priority>4</priority> <description/> <type>0</type> <manual_close>0</manual_close> <dependencies/> <tags/> </trigger> <trigger> <expression>{Template TCP Connection Status:tcp.status[timewait].last()}>10000</expression> <recovery_mode>0</recovery_mode> <recovery_expression/> <name>There are too many TCP TIME_WAIT status</name> <correlation_mode>0</correlation_mode> <correlation_tag/> <url/> <status>0</status> <priority>4</priority> <description/> <type>0</type> <manual_close>0</manual_close> <dependencies/> <tags/> </trigger> </triggers> <graphs> <graph> <name>TCP Status</name> <width>900</width> <height>200</height> <yaxismin>0.0000</yaxismin> <yaxismax>100.0000</yaxismax> <show_work_period>1</show_work_period> <show_triggers>1</show_triggers> <type>0</type> <show_legend>1</show_legend> <show_3d>0</show_3d> <percent_left>0.0000</percent_left> <percent_right>0.0000</percent_right> <ymin_type_1>0</ymin_type_1> <ymax_type_1>0</ymax_type_1> <ymin_item_1>0</ymin_item_1> <ymax_item_1>0</ymax_item_1> <graph_items> <graph_item> <sortorder>0</sortorder> <drawtype>0</drawtype> <color>C80000</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[closed]</key> </item> </graph_item> <graph_item> <sortorder>1</sortorder> <drawtype>0</drawtype> <color>00C800</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[closewait]</key> </item> </graph_item> <graph_item> <sortorder>2</sortorder> <drawtype>0</drawtype> <color>0000C8</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[closing]</key> </item> </graph_item> <graph_item> <sortorder>3</sortorder> <drawtype>0</drawtype> <color>C800C8</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[established]</key> </item> </graph_item> <graph_item> <sortorder>4</sortorder> <drawtype>0</drawtype> <color>00C8C8</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[finwait1]</key> </item> </graph_item> <graph_item> <sortorder>5</sortorder> <drawtype>0</drawtype> <color>C8C800</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[finwait2]</key> </item> </graph_item> <graph_item> <sortorder>6</sortorder> <drawtype>0</drawtype> <color>C8C8C8</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[lastack]</key> </item> </graph_item> <graph_item> <sortorder>7</sortorder> <drawtype>0</drawtype> <color>960000</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[listen]</key> </item> </graph_item> <graph_item> <sortorder>8</sortorder> <drawtype>0</drawtype> <color>009600</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[synrecv]</key> </item> </graph_item> <graph_item> <sortorder>9</sortorder> <drawtype>0</drawtype> <color>000096</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[synsent]</key> </item> </graph_item> <graph_item> <sortorder>10</sortorder> <drawtype>0</drawtype> <color>960096</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[timewait]</key> </item> </graph_item> </graph_items> </graph> </graphs> </zabbix_export>