學習OpenStack之（6）：Neutron 深入學習之 OVS + GRE 之 Compute node 篇

0.環境

硬件環境見上一篇博客：學習OpenStack之（5）：在Mac上部署Juno版本OpenStack 四節點環境

OpenStack網絡配置：一個tenant， 2個虛機

• Type driver： GRE， Mechanism driver： OVS
• 一個public network： ext-net 和相應的subnet ext-subnet
• 一個VM network：demo-net 和相應的subnet：demo-subnet
• 一個router連接ext-subnet和demo-subnet

 

1、Compute 節點上networking組件

下面會用到OVS的兩個重要命令：

• ovs-vsctl： 查詢和更新ovs-vswitchd的配置
• ovs-ofctl： 查詢和控制OpenFlow交換機和控制器

首先查詢Compute節點上ovs-vswitchd的配置的配置：

root@compute1:/var/lib/nova# ovs-vsctl show
205a13a2-1ad6-4ae0-8c84-abed97444aa9
    Bridge br-int //OVS integration 橋 br-int
        fail_mode: secure
        Port "qvo37b25c08-e8" //端口，用來連接一個虛機網卡的TAP設備所連接的linux bridge
            tag: 1
            Interface "qvo37b25c08-e8"
        Port patch-tun //端口，用來連接橋br-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int} //和橋 br-tun上的patch-int是對等端口
        Port br-int
            Interface br-int
                type: internal
        Port "qvo155845ae-5e" //端口，用來連接另一個虛機網卡的TAP設備所連接的linux bridge
            tag: 1
            Interface "qvo155845ae-5e"
    Bridge br-tun //OVS Tunnel 橋br-tun
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int //端口patch-int，用來連接橋br-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a000115" //端口，連接GRE Tunnel
            Interface "gre-0a000115"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="10.0.1.31", out_key=flow, remote_ip="10.0.1.21"}
    ovs_version: "2.0.2" //GRE Tunnel是點到點之間建立的，這頭的IP為10.0.1.31，那頭的IP地址為 10.0.1.21

繼續看橋 br-tun：

root@compute1:/var/lib/nova# ovs-ofctl show br-tun
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000f6b428614747
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(patch-int): addr:3e:7b:d5:fa:26:8d //端口 patch-int的ID 是 1
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 2(gre-0a000115): addr:2a:26:b2:99:f3:5a //端口 gre-0a000115的ID 是 2
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-tun): addr:f6:b4:28:61:47:47
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

每個虛機有個虛機網卡 eth0，eth0和host上的一個TAP設備連接，該TAP設備直接掛載在一個Linux Bridge上，該Linux Bridge和OVS integration bridge br-int相連。其實理想情況下，TAP設備能和OVS Integration Bridge 直接相連就好了，但是，因為OpenStack實現Security Group的需要，這里要多加一層Linux bridge。OpenStack使用Linux TAP設備上的iptables來實現Security Group規則，而OVS不支持直接和br-int橋相連的TAP設備上的iptables。通過查看虛機的libvirt XML定義文件 /var/lib/nova/instances/<instance-id>/libvirt.xml可以看出來虛機所連接的TAP設備：

<interface type="bridge">
      <mac address="fa:16:3e:fe:c7:87"/> //
      <source bridge="qbr37b25c08-e8"/> //虛機TAP設備所掛接的linux bridge
      <target dev="tap37b25c08-e8” /> //虛機所連接的interface 
</interface>

通過以上信息，我們可以畫出compute 節點上的網絡組建圖：

2. Neutron使用TAP設備的iptables來實現Security groups

查看第一個虛機的TAP設備上的iptables：

root@compute1:/var/lib/nova# iptables -S | grep tap37b25c08-e8
-A neutron-openvswi-FORWARD -m physdev --physdev-out tap37b25c08-e8 --physdev-is-bridged -j neutron-openvswi-sg-chain
-A neutron-openvswi-FORWARD -m physdev --physdev-in tap37b25c08-e8 --physdev-is-bridged -j neutron-openvswi-sg-chain
-A neutron-openvswi-INPUT -m physdev --physdev-in tap37b25c08-e8 --physdev-is-bridged -j neutron-openvswi-o37b25c08-e
-A neutron-openvswi-sg-chain -m physdev --physdev-out tap37b25c08-e8 --physdev-is-bridged -j neutron-openvswi-i37b25c08-e
-A neutron-openvswi-sg-chain -m physdev --physdev-in tap37b25c08-e8 --physdev-is-bridged -j neutron-openvswi-o37b25c08-e

OpenStack Neutron在neutron-openvswi-sg-chain上實現security groups。在使用默認security group的情況下：

• neutron-openvswi-o37b25c08-e 控制從虛機出去的traffic

              -A neutron-openvswi-o37b25c08-e -p udp -m udp --sport 68 --dport 67 -j RETURN 
             
-A neutron-openvswi-o37b25c08-e -j neutron-openvswi-s37b25c08-e 
             
-A neutron-openvswi-o37b25c08-e -p udp -m udp --sport 67 --dport 68 -j DROP 
             
-A neutron-openvswi-o37b25c08-e -m state --state INVALID -j DROP 
             
-A neutron-openvswi-o37b25c08-e -m state --state RELATED,ESTABLISHED -j RETURN 
             
-A neutron-openvswi-o37b25c08-e -j RETURN 
             
-A neutron-openvswi-o37b25c08-e -j neutron-openvswi-sg-fallback 
            

• neutron-openvswi-i37b25c08-e 控制進入虛機的traffic

-A neutron-openvswi-i37b25c08-e -m state --state INVALID -j DROP
-A neutron-openvswi-i37b25c08-e -m state --state RELATED,ESTABLISHED -j RETURN
-A neutron-openvswi-i37b25c08-e -s 10.0.0.116/32 -p udp -m udp --sport 67 --dport 68 -j RETURN
-A neutron-openvswi-i37b25c08-e -p tcp -m tcp --dport 22 -j RETURN
-A neutron-openvswi-i37b25c08-e -p icmp -j RETURN
-A neutron-openvswi-i37b25c08-e -m set --match-set IPv48c0dc337-0a6d-4ad7-9 src -j RETURN
-A neutron-openvswi-i37b25c08-e -j neutron-openvswi-sg-fallback

使用下面的命令來添加一條secrutiy group 規則來允許使用TCP 22端口：

neutron security-group-rule-create --protocol tcp --port-range-min 22 --port-range-max 22 --direction ingress default

那么該TAP設備的iptables會出現下面的變化：

root@compute1:/var/lib/nova# iptables -S | grep 22
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A neutron-openvswi-i155845ae-5 -p tcp -m tcp --dport 22 -j RETURN
-A neutron-openvswi-i155845ae-5 -p tcp -m tcp --dport 22 -j RETURN
-A neutron-openvswi-i37b25c08-e -p tcp -m tcp --dport 22 -j RETURN
-A neutron-openvswi-i37b25c08-e -p tcp -m tcp --dport 22 -j RETURN

3. OVS integration 橋 br-int添加和刪除traffic的VLAN ID

每一個使用 neutron net-create 命令創建的network都有一個新的 VLAN ID.本例中因為只有一個network，所以VLAN ID是1，見ovsctl-vsctl show命令中的port tag值。

4. OVS Tunnel 橋 br-tun 處理 VLAN ID 和 Tunnel ID的轉化

從以下OpenFlow rule tables可見兩種ID的處理過程：

root@compute1:/var/lib/nova# ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=11509.036s, table=0, n_packets=1059, n_bytes=116533, idle_age=740, priority=1,in_port=1 actions=resubmit(,2) //從端口1及patch-int進來的traffic會被重新執行table 2的rule
 cookie=0x0, duration=2089.491s, table=0, n_packets=1082, n_bytes=115494, idle_age=741, priority=1,in_port=2 actions=resubmit(,3) //從端口2 即 gre 端口進來的traffic重新執行table 3
 cookie=0x0, duration=11508.939s, table=0, n_packets=5, n_bytes=390, idle_age=11500, priority=0 actions=drop
 cookie=0x0, duration=11508.84s, table=2, n_packets=955, n_bytes=106446, idle_age=741, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20) //重新執行table 20的rule
 cookie=0x0, duration=11508.745s, table=2, n_packets=104, n_bytes=10087, idle_age=740, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
 cookie=0x0, duration=2260.307s, table=3, n_packets=1082, n_bytes=115494, idle_age=741, priority=1,tun_id=0x1 actions=mod_vlan_vid:1,resubmit(,10) //從neutron node來的traffic，打上VLAN ID 1，重新執行table 10的 rule
 cookie=0x0, duration=11508.646s, table=3, n_packets=15, n_bytes=1274, idle_age=2098, priority=0 actions=drop
 cookie=0x0, duration=11508.495s, table=4, n_packets=0, n_bytes=0, idle_age=11508, priority=0 actions=drop
 cookie=0x0, duration=11508.293s, table=10, n_packets=1082, n_bytes=115494, idle_age=741, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1 //學習規則 table 20，從port 1 即 patch-int發出
 cookie=0x0, duration=11508.093s, table=20, n_packets=0, n_bytes=0, idle_age=11508, priority=0 actions=resubmit(,22) //重新執行table 22的rule
 cookie=0x0, duration=2260.372s, table=22, n_packets=77, n_bytes=7817, idle_age=740, hard_age=2089, dl_vlan=1 actions=strip_vlan,set_tunnel:0x1,output:2,output:2 //去掉VLAN ID，打上TUNNEL ID 1 即 neutron 節點的TUNNEL ID，從端口2 即 gre 端口發出
 cookie=0x0, duration=11507.901s, table=22, n_packets=27, n_bytes=2270, idle_age=1664, priority=0 actions=drop

下一節將neutron節點。