1. 環境
一台物理服務器 9.115.251.86，上面創建兩個虛機，每個虛機兩個網卡：
- vm1：eth0 - 9.*.*.232 eth1：10.0.0.14
- vm2: eth0 - 9.8.*.219 eth1:10.0.0.10，上面運行DHCP Agent，管理 dnsmasq，提供 DHCP 服務
兩塊 eth1 連到物理機上的一個 bridge 上：
bridge name bridge id STP enabled interfaces mgtbr0 8000.fa8013216b56 no tap0 vnet1 vnet12 vnet13 vnet14 vnet16 vnet18 vnet2 vnet5
2. 錯誤
vm1 上創建一個虛機 vmchild1，啟動時發出 BOOTP 請求去向 DHCP Agent 獲取 IP 地址。DHCP Agent 管理的 dnsmasq 可以正常收到 BOOTP 請求，而且發回了請求，但是 vm1 無法收到。
vm2 上 tcpdump：
15:30:14.135874 IP (tos 0x0, ttl 64, id 46594, offset 0, flags [none], proto UDP (17), length 401) 10.0.0.10.40589 > 10.0.0.14.8472: OTV, flags [I] (0x08), overlay 0, instance 1027 IP (tos 0xc0, ttl 64, id 57625, offset 0, flags [none], proto UDP (17), length 351) 50-0-0-10.static.sonic.net.bootps > 50-0-0-15.static.sonic.net.bootpc: BOOTP/DHCP, Reply, length 323, xid 0x11f7631f, secs 60, Flags [none] Your-IP 50-0-0-15.static.sonic.net Server-IP 50-0-0-10.static.sonic.net Client-Ethernet-Address fa:16:3e:ba:43:4c (oui Unknown) Vendor-rfc1048 Extensions Magic Cookie 0x63825363 DHCP-Message Option 53, length 1: Offer Server-ID Option 54, length 4: 50-0-0-10.static.sonic.net Lease-Time Option 51, length 4: 86400 RN Option 58, length 4: 43200 RB Option 59, length 4: 75600 Subnet-Mask Option 1, length 4: 255.255.255.0 BR Option 28, length 4: 50-0-0-255.static.sonic.net Domain-Name Option 15, length 14: "openstacklocal" Domain-Name-Server Option 6, length 4: 50-0-0-10.static.sonic.net Default-Gateway Option 3, length 4: 50-0-0-1.static.sonic.net Classless-Static-Route Option 121, length 13: (60.0.0.0/24:0.0.0.0),(default:50-0-0-1.static.sonic.net) 15:30:14.136118 IP (tos 0xc0, ttl 64, id 10816, offset 0, flags [none], proto ICMP (1), length 429) 10.0.0.100 > 10.0.0.10: ICMP host 10.0.0.14 unreachable - admin prohibited, length 409 IP (tos 0x0, ttl 63, id 46594, offset 0, flags [none], proto UDP (17), length 401) 10.0.0.10.40589 > 10.0.0.14.8472: OTV, flags [I] (0x08), overlay 0, instance 1027
而且發現另外的問題：
root@controller:~/s1# ping 10.0.0.13 PING 10.0.0.13 (10.0.0.13) 56(84) bytes of data. 64 bytes from 10.0.0.13: icmp_seq=1 ttl=64 time=0.630 ms From 10.0.0.13: icmp_seq=2 Redirect Host(New nexthop: 10.0.0.13) 64 bytes from 10.0.0.13: icmp_seq=2 ttl=64 time=0.628 ms From 10.0.0.13: icmp_seq=3 Redirect Host(New nexthop: 10.0.0.13)
3.定位
初步推斷是某處防火牆阻止了 BOOTP 的包，於是將 vm1，vm2 上的防火牆全部關掉，無效。最后，查看物理機 iptables：
[root@rh65 ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT icmp -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:domain ACCEPT udp -- anywhere anywhere state NEW udp dpt:domain ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ftp ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:16509 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:16514 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:http ACCEPT tcp -- anywhere anywhere state NEW tcp dpts:vnc-server:cvsup REJECT all -- anywhere anywhere reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT icmp -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
發現 FORWARD 阻止了 ICMP 相關的包。
4. 解決
在物理機上運行 service iptables stop 將 iptables 關閉，問題解決。