CentOS 6.9下iptables通過raw表實現日志輸出和調試

說明：iptables調試的最好方式應該是輸出日志了。並且iptables有個raw的表，優先級別最好，且調試時針對icmp協議（ping）進行，那么日志輸出就是整條鏈路串起來輸出的，非常的清晰。

前提：

必須配置了日志輸出，參考：http://www.cnblogs.com/EasonJim/p/8413715.html

背景：

在配置folsom版openstack的quantum時出現vm無法ping通外網的問題，經過抓包分析確定問題是iptables中的snat規則不生效，需要調試iptables定位下有問題的iptables規則。 iptables有5個鏈: PREROUTING, INPUT, FORWARD, OUTPUT, POSTROUTING，4個表: filter, nat, mangle, raw，其工作流在下圖表示的很清楚。 

從圖中可見raw擁有最高的優先級，raw表使用PREROUTING和OUTPUT兩個鏈，因此raw可以覆蓋所有包。在raw表中支持一個特殊的目標:TRACE，使內核記錄下每條匹配該包的對應iptables規則信息。使用raw表內的TRACE target即可實現對iptables規則的跟蹤調試。

配置：

假設需要對ipv4的ICMP包進行跟蹤調試，抓取所有流經本機的ICMP包

iptables -t raw -A OUTPUT -p icmp -j TRACE
iptables -t raw -A PREROUTING -p icmp -j TRACE

加載對應內核模組

modprobe ipt_LOG

調試信息記錄在/var/log/kern.log文件（也可以查看/var/log/messages）。

示例調試：

在vm內對外部作ping操作，vm的ip為10.0.0.4

[root@10-0-0-4 ~]# ping -c 1 192.168.0.19
PING 192.168.0.19 (192.168.0.19)56(84) bytes of data.
 
--- 192.168.0.19 ping statistics ---1 packets transmitted, 0 received, 100% packet loss, time 0ms

在/var/log/kern.log中的對應調試信息如下

Apr 1811:50:23 openstack-network kernel: [1038991.870882] TRACE: raw:PREROUTING:policy:2IN=tap5c42978b-ac OUT= MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=64ID=0 DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=1
Apr 1811:50:23 openstack-network kernel: [1038991.870902] TRACE: nat:PREROUTING:rule:1IN=tap5c42978b-ac OUT= MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=64ID=0 DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=1
Apr 1811:50:23 openstack-network kernel: [1038991.870909] TRACE: nat:quantum-l3-agent-PREROUTING:return:4IN=tap5c42978b-ac OUT= MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=64ID=0 DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=1
Apr 1811:50:23 openstack-network kernel: [1038991.870915] TRACE: nat:PREROUTING:policy:2IN=tap5c42978b-ac OUT= MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=64ID=0 DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=1
Apr 1811:50:23 openstack-network kernel: [1038991.870938] TRACE: filter:FORWARD:rule:1IN=tap5c42978b-ac OUT=br-ex MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=63ID=0 DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=1
Apr 1811:50:23 openstack-network kernel: [1038991.870944] TRACE: filter:quantum-filter-top:rule:1IN=tap5c42978b-ac OUT=br-ex MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=63ID=0 DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=1
Apr 1811:50:23 openstack-network kernel: [1038991.870950] TRACE: filter:quantum-l3-agent-local:return:1IN=tap5c42978b-ac OUT=br-ex MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=63ID=0 DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=1
Apr 1811:50:23 openstack-network kernel: [1038991.870957] TRACE: filter:quantum-filter-top:return:2IN=tap5c42978b-ac OUT=br-ex MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=63ID=0 DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=1
Apr 1811:50:23 openstack-network kernel: [1038991.870962] TRACE: filter:FORWARD:rule:2IN=tap5c42978b-ac OUT=br-ex MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=63ID=0 DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=1
Apr 1811:50:23 openstack-network kernel: [1038991.870969] TRACE: filter:quantum-l3-agent-FORWARD:return:1IN=tap5c42978b-ac OUT=br-ex MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=63ID=0 DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=1
Apr 1811:50:23 openstack-network kernel: [1038991.870974] TRACE: filter:FORWARD:policy:3IN=tap5c42978b-ac OUT=br-ex MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=63ID=0 DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=1
Apr 1811:50:23 openstack-network kernel: [1038991.870979] TRACE: nat:POSTROUTING:rule:1IN= OUT=br-ex SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=63ID=0 DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=1
Apr 1811:50:23 openstack-network kernel: [1038991.870985] TRACE: nat:quantum-l3-agent-POSTROUTING:rule:1IN= OUT=br-ex SRC=10.0.0.4 DST=192.168.0.19 LEN=84TOS=0x00 PREC=0x00 TTL=63ID=0 DF PROTO=ICMP TYPE=8CODE=0ID=28976SEQ=1

可見數據包流在nat表的quantum-l3-agent-POSTROUTING的第一條規則處被截斷了，查看iptables中的nat表的規則如下

*nat
:PREROUTING ACCEPT [99:21975]
:INPUT ACCEPT [74:20608]
:OUTPUT ACCEPT [181:30548]
:POSTROUTING ACCEPT [26:13022]
:quantum-l3-agent-OUTPUT - [0:0]
:quantum-l3-agent-POSTROUTING - [0:0]
:quantum-l3-agent-PREROUTING - [0:0]
:quantum-l3-agent-float-snat - [0:0]
:quantum-l3-agent-snat - [0:0]
:quantum-postrouting-bottom - [0:0]-A PREROUTING -j quantum-l3-agent-PREROUTING
-A OUTPUT -j quantum-l3-agent-OUTPUT
-A POSTROUTING -j quantum-l3-agent-POSTROUTING
-A POSTROUTING -j quantum-postrouting-bottom
-A quantum-l3-agent-OUTPUT -d 192.168.0.16/32-j DNAT --to-destination 10.0.0.4
-A quantum-l3-agent-OUTPUT -d 192.168.0.17/32-j DNAT --to-destination 10.0.0.3
-A quantum-l3-agent-POSTROUTING !-i qg-91757ded-c4 !-o qg-91757ded-c4 -m conntrack !--ctstate DNAT -j ACCEPT
-A quantum-l3-agent-POSTROUTING -s 10.0.0.0/24-d 192.168.1.1/32-j ACCEPT
-A quantum-l3-agent-PREROUTING -d 169.254.169.254/32-p tcp -m tcp --dport80-j DNAT --to-destination 192.168.1.1:8775-A quantum-l3-agent-PREROUTING -d 192.168.0.16/32-j DNAT --to-destination 10.0.0.4
-A quantum-l3-agent-PREROUTING -d 192.168.0.17/32-j DNAT --to-destination 10.0.0.3
-A quantum-l3-agent-float-snat -s 10.0.0.4/32-j SNAT --to-source 192.168.0.16
-A quantum-l3-agent-snat -j quantum-l3-agent-float-snat
-A quantum-l3-agent-snat -s 10.0.0.0/24-j SNAT --to-source 192.168.0.15
-A quantum-postrouting-bottom -j quantum-l3-agent-snat
COMMIT

確定有問題的規則為

-A quantum-l3-agent-POSTROUTING !-i qg-91757ded-c4 !-o qg-91757ded-c4 -m conntrack !--ctstate DNAT -j ACCEPT

把這條規則刪掉后重啟iptables，vm能順利連接外網，問題解決。

 

參考：

http://blog.51cto.com/flymanhi/1276331（以上內容轉自此篇文章）

http://backreference.org/2010/06/11/iptables-debugging/