問題的提出
最近在准備客戶端的新版本，在內部灰度過程中，發現一類奇怪的 dump，通過查看日志和堆棧，可以確定是因為每次連上后台就被后台斷開了、導致多次重連后隨機發生的崩潰。dump 和日志都無法提供進一步的信息來定位問題，本地又不復現，也沒有辦法去聯系用戶查看現場 (windows 服務崩潰后自動重啟，用戶不感知)。於是想到能不能自己制造這樣的場景 —— 當連接建立后立即斷開該連接 —— 看是否會復現崩潰。
問題的解決
tcpview
在 windows 上最直觀的解決方案就是手動斷開連接啦，拿出 sysinternal 工具集，翻出 tcpview，就可以看到系統上所有的 tcp 連接了：
除了能看到建立的 tcp 連接所屬進程、本地地址/端口號、遠端地址/端口號外、連接狀態外，還可以看到一些連接上的統計信息，如收發包數和字節數等。高亮的那一行就是我想要殺掉的連接。在 tcpview 里殺連接很簡單，直接右鍵菜單 ‘Close Connection’ 即可。但是這樣做的問題是，每次從看到連接到殺死連接要經歷一定時間 (手動操作)，甚至進程日志已經顯示連接建立了，tcpview 還沒有刷出來，總而言之就是一個字 —— 慢，殺了十幾次，掛上調試器的進程紋絲不動，一點要崩潰的跡象也沒有 (關鍵是還手疼)，汗~
tcpkill
都 2020 年了，自動化工具用起來，查了一下，tcpview 除了界面外沒有提供類似命令行的功能，於是只能在網上搜 “什么命令能殺掉 tcp 連接” 了，百度到一個 tcpkill，這個命令是原生於 linux 的，需要先安裝 dsniff 工具包
$ sudo yum install dsniff
安裝成功后就可以實操一下了
$ tcpkill Version: 2.4 Usage: tcpkill [-i interface] [-1..9] expression
help 和 man 都太過簡單，其實重點就是最后這個參數 expression，貌似是使用和 tcpdump 相同的格式，為了驗證 tcpkill，我先搭建了一個簡單的測試環境：
$ nc -4 -l -p 5555
使用 nc 創建一個在 5555 端口監聽的進程；
$ nc -4 localhost 5555 -p 6666
在另外一個終端中創建一個進程去連接 5555 端口，它自己的端口是 6666；
$ netstat -antp (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name …… tcp 0 0 127.0.0.1:5555 127.0.0.1:6666 ESTABLISHED 2451/nc tcp 0 0 127.0.0.1:6666 127.0.0.1:5555 ESTABLISHED 2470/nc ……
可以通過 netstat 命令查看到建立的這個連接及其端口號。下面用 tcpkill 殺掉這個連接，這里使用指定 6666 端口的方式：
$ sudo tcpkill -i lo port 6666 tcpkill: listening on lo [port 6666] 127.0.0.1:6666 > 127.0.0.1:5555: R 18446744073486680909:18446744073486680909(0) win 0 127.0.0.1:6666 > 127.0.0.1:5555: R 18446744073486681251:18446744073486681251(0) win 0 127.0.0.1:6666 > 127.0.0.1:5555: R 18446744073486681935:18446744073486681935(0) win 0 127.0.0.1:5555 > 127.0.0.1:6666: R 18446744071881223293:18446744071881223293(0) win 0 127.0.0.1:5555 > 127.0.0.1:6666: R 18446744071881223635:18446744071881223635(0) win 0 127.0.0.1:5555 > 127.0.0.1:6666: R 18446744071881224319:18446744071881224319(0) win 0
打印了一堆莫名其妙的信息，看 netstat 輸出的話，那個連接還是 ESTABLISHED 狀態，但是通過在控制台輸入一些字符 (例如 abc) 來發送數據后，連接中斷、進程退出
$ nc -4 localhost 5555 -p 6666 hello world abc Ncat: Connection reset by peer.
在 5555 端口監聽的進程由於連接中斷也自動退出了
$ nc -4 -l -p 5555 hello world abc
此時再看 netstat 輸出，就看不到這條連接的相關信息了。看相關文章，貌似是這個命令向連接發送了偽造的 rst 包，所以只有當下次客戶端再請求時，才會發現連接已經中斷了。而且 tcpkill 好像會一直運行，只要它發現在 6666 這個端口建立了連接，就會去嘗試中斷。雖然后面這個特性挺好，但是連接只有在下一次發送數據時才能檢測到中斷這事，實效性差那么點兒意思；最麻煩的是我在 windows 的 msys2 環境中，沒有這個命令可用，看來這個命令依賴的一些 linux 底層機制在 win32 上不太好實現，於是果斷放棄。
cports
下面百度的重點就放在了 “windows 上可以殺掉 tcp 連接的命令” 了，結果還真被找到一個 —— CurrPorts，它本身是個 UI 工具，界面和 tcpview 很類似：
輸出的信息大同小異，都是進程、協議、本地地址/端口、遠程地址/端口、連接狀態等 (其實還少了一些連接上的統計信息)。然后關閉連接也是通過選中項目后右鍵菜單來實現的：
光看菜單的話，感覺比 tcpview 功能豐富多了，比如光選項就有這么多：
確實比 tcpview 要好用一些，但是使用右鍵菜單來關閉連接，貌似和之前沒有多大分別呢 (雖然可以使用 Ctrl+T 快捷方式)。不要急，下面着重說一下 CurrPorts 的命令行參數，這是區別於 tcpview 的一大優勢：
| /stext <Filename> | Save the list of all opened TCP/UDP ports into a regular text file. |
| /stab <Filename> | Save the list of all opened TCP/UDP ports into a tab-delimited text file. |
| /scomma <Filename> | Save the list of all opened TCP/UDP ports into a comma-delimited text file. |
| /stabular <Filename> | Save the list of all opened TCP/UDP ports into a tabular text file. |
| /shtml <Filename> | Save the list of all opened TCP/UDP ports into HTML file (Horizontal). |
| /sverhtml <Filename> | Save the list of all opened TCP/UDP ports into HTML file (Vertical). |
| /sxml <Filename> | Save the list of all opened TCP/UDP ports to XML file. |
| /CaptureTime <Milliseconds> | Specifies the capture time in milliseconds for the save command-line options (/stext, /stab, /scomma, and so on...) Example: cports.exe /RunAsAdmin /scomma c:\temp\ports1.csv /CaptureTime 15000 |
| /RunAsAdmin | Runs CurrPorts as Administrator. |
| /sort <column> | This command-line option can be used with other save options for sorting by the desired column. If you don't specify this option, the list is sorted according to the last sort that you made from the user interface. The <column> parameter can specify the column index (0 for the first column, 1 for the second column, and so on) or the name of the column, like "Remote Port" and "Remote Address". You can specify the '~' prefix character (e.g: "~Remote Address") if you want to sort in descending order. You can put multiple /sort in the command-line if you want to sort by multiple columns. Examples: |
| /nosort | When you specify this command-line option, the list will be saved without any sorting. |
| /filter <filter string> | Start CurrPorts with the specified filters. If you want to specify more than one filter, use the ';' character as a delimiter. |
| /cfg <cfg filename> | Start CurrPorts with the specified config file. |
| /MarkPorts /DisplayUdpPorts /DisplayTcpPorts /DisplayClosedPorts /MarkNewModifiedPorts /SortOnAutoRefresh /AlwaysOnTop /AskBefore /DisplayIPv6Ports /DisplayListening /DisplayEstablished /DisplayNoState /DisplayNoRemoteIP /ResolveAddresses /RememberLastFilter /DisplayPortInAddress /AutoRefresh, /ShowInfoTip /TrayIcon /TrayIconOneClick /StartAsHidden /LogChanges /LogFilename /DisabledFilters /AddExportHeaderLine |
You can use all these parameters to control the options that are available under the Options and View menus. For example, if you want to start CurrPorts with 'Display UDP Ports' turned off and 'Display Closed' turned on: cports.exe /DisplayUdpPorts 0 /DisplayClosedPorts 1 You can also use these parameters in conjunction with all save parameters. For example: If you want to save into tab-delimited file only the UDP ports: |
看了一圈兒，好像都是將結果保存到文件的一些選項，還好下面有一段話是專門說明如何關閉連接的：
Closing a Connection From Command-Line
Starting from version 1.09, you can close one or more connections from command-line, by using /close parameter.
The syntax of /close command:
/close <Local Address> <Local Port> <Remote Address> <Remote Port> {Process Name/ID}
For each parameter, you can specify "*" in order to include all ports or addresses. The process name is an optional parameter. If you specify a process, only the ports of the specified process will be closed.
Examples:
Close all connections with remote port 80 and remote address 192.168.1.10:
/close * * 192.168.1.10 80
Close all connections with remote port 80 (for all remote addresses):
/close * * * 80
Close all connections to remote address 192.168.20.30:
/close * * 192.168.20.30 *
Close all connections with local port 80:
/close * 80 * *
Close all connections of Firefox with remote port 80:
/close * * * 80 firefox.exe
Close all connections of the process that its ID is 3276:
/close * * * * 3276
給出了豐富的示例，例如針對我們的場景，可以這樣調用 cports (UI 叫 CurrPorts，但命令名為 cports.exe)：
cports /close * * xxx.xxx.xx.xx 3570 gdphost.exe
其中 xxx 部分是連接的遠端 IP 地址，3570 是遠端端口，gdphost.exe 是發起連接的進程名。當 cmd 以管理員權限運行，上面的調用是可以殺死連接的，但是在 msys2 bash (其實就是 git bash 啦) 中運行卻直接啟動了 CurrPorts 工具的 UI 界面。問題可能出在 /close 被 bash 作了轉義識別上，用雙引號將它們包含一下：
cports "/close" "*" "*" "xxx.xxx.xx.xx" "3570" "gdphost.exe"
結果還是不行，經過一翻研究，改成下面這樣就可以了：
cports "//close" "*" "*" "xxx.xxx.xx.xx" "3570" "gdphost.exe"
在 msys2 bash 中要對 win32 命令的選項開始符 '/' 使用轉義符前綴，否則 bash 會認為 /close 是一個目錄(?)而非參數，從而進行某種轉換(?)。又研究了一下雙引號的作用，發現對於星號還是必需加上的，其它的參數可以不加，於是最后版本就成了這樣：
cports "//close" "*" "*" xxx.xxx.xx.xx 3570 gdphost.exe
注意 cports 所在目錄我是添加到了 Path 環境變量，所以可以這樣直接寫命令，否則必需提供 cports 的完整路徑。另外這個 bash 也必需以管理員權限啟動，不然命令雖然可以返回，但是沒有發生任何實質性影響。
完整腳本
有了 cports 的底層支持，我們就可以這樣寫腳本來自動斷開連接“偽造”事發現場啦：
1 #! /bin/sh 2 while true 3 do 4 n=$(netstat -ano | grep 3570 | wc -l) 5 if [ $n -gt 0 ]; then 6 cports "//close" "*" "*" "*" 3570 gdphost.exe 7 echo "close connection" 8 else 9 echo "no connection find" 10 fi 11 done
加了一點簡單的邏輯，在運行循環中首先通過 netstat 判斷有沒有在 3570 端口建立的連接，如果有就調用 cports 去斷開；如果沒有就繼續循環，直到 Ctrl+C 結束。腳本運行起來后，結合圖形界面、可以看到這個進程到后台的連接在不停的斷開重連，下面是觀察到腳本的一些輸出 (內容較多、展開慎重)：
$ ./kill.sh close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find ^C
錄取了大概一千行的輸出，平均每 80 行可以觀察到一次中斷連接的記錄，也就是說 netstat 運行 80 次左右才能輪到一次 cports，這和進程在連接中斷后 5 秒進行第一次重試有關 (通過換算，netstat 一秒運行了 16 次?)，而為了“及時”殺掉進程這里也沒有采用 sleep 去避免忙等待 (其實可以等待 4 秒)。
結語
今天通過一個實際場景來研究了一下如何使用 shell + cports 不斷的殺死某個連接、進而構造一個 bug 復現的場景。雖然腳本寫的很漂亮，但遺憾的是這個 bug 未能復現 (淚奔~)，我用調試器掛上 win32 進程跑了四個多小時也沒有出現崩潰。再仔細對比 log 輸出，發現偽造的場景下進程還是可以有機會輸出更多信息，看來“殺”的還是不夠快啊；於是我將 netstat 檢測 3570 端口是否存在這步去掉了，直接在循環里調用 cports 不斷的殺連接，這應該比之前快了吧，但是還是沒有發生崩潰；所以我感覺下一步只能用 c++ 寫個程序，模擬在 3570 端口偵聽、並在連接一上來的時候就 close connection 試試了 (需要設置 host 以便將域名指向本地啟動的這個模擬程序)。
雖然沒能復現 bug，但是無意間得到了 cports 這個寶貝，另外它的作者 nir sofer 也是一個 windows 工具小達人，制作了一系列 win32 工具供網友免費下載使用，有興趣的讀者可以去他的網站上翻翻。
使用 shell 腳本建立 tcp 連接可以參考我之前寫的這篇文章：《用 shell 腳本做 tcp 協議模擬 》。
參考
[1]. Windows Sysinternals
[2]. Tcpkill
[3]. How to kill a particular tcp connection in windows?
[4]. CPorts
[5]. NirSoft