六、Libvirt
對於Libvirt，在啟動虛擬機之前，首先需要define虛擬機，是一個XML格式的文件
列出所有的Instance
# virsh list
Id Name State
----------------------------------------------------
10 instance-00000006 running
# virsh dumpxml instance-00000006
<domain type='kvm' id='10'>
<name>instance-00000006</name>
<uuid>73b896bb-7c7d-447e-ab6a-c4089532f003</uuid>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<vcpu placement='static'>1</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<sysinfo type='smbios'>
<system>
<entry name='manufacturer'>OpenStack Foundation</entry>
<entry name='product'>OpenStack Nova</entry>
<entry name='version'>2014.1.1</entry>
<entry name='serial'>80590690-87d2-e311-b1b0-a0481cabdfb4</entry>
<entry name='uuid'>73b896bb-7c7d-447e-ab6a-c4089532f003</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'>
<timer name='pit' tickpolicy='delay'/>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/var/lib/nova/instances/73b896bb-7c7d-447e-ab6a-c4089532f003/disk'/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<controller type='usb' index='0'>
<alias name='usb0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<interface type='bridge'>
<mac address='fa:16:3e:ae:f4:17'/>
<source bridge='qbrc51a349e-87'/>
<target dev='tapc51a349e-87'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='file'>
<source path='/var/lib/nova/instances/73b896bb-7c7d-447e-ab6a-c4089532f003/console.log'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<serial type='pty'>
<source path='/dev/pts/20'/>
<target port='1'/>
<alias name='serial1'/>
</serial>
<console type='file'>
<source path='/var/lib/nova/instances/73b896bb-7c7d-447e-ab6a-c4089532f003/console.log'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<input type='tablet' bus='usb'>
<alias name='input0'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0' keymap='en-us'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</memballoon>
</devices>
<seclabel type='dynamic' model='apparmor' relabel='yes'>
<label>libvirt-73b896bb-7c7d-447e-ab6a-c4089532f003</label>
<imagelabel>libvirt-73b896bb-7c7d-447e-ab6a-c4089532f003</imagelabel>
</seclabel>
</domain>
我們發現里面定義了虛擬化類型kvm, vcpu, memory, disk, pty等，需要注意的是network，是一個tap device，attach到了qbr上。
虛擬化有很多種類型，參考下面的文章
當然虛擬機啟動了之后，通過進程的查看，便能看到復雜無比的參數
# ps aux | grep instance-00000006
libvirt+ 22200 6.3 0.4 5464532 282888 ? Sl 09:51 0:09 qemu-system-x86_64 -enable-kvm -name instance-00000006 -S -machine pc-i440fx-trusty,accel=kvm,usb=off -cpu SandyBridge,+erms,+smep,+fsgsbase,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 73b896bb-7c7d-447e-ab6a-c4089532f003 -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=2014.1.1,serial=80590690-87d2-e311-b1b0-a0481cabdfb4,uuid=73b896bb-7c7d-447e-ab6a-c4089532f003 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000006.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/73b896bb-7c7d-447e-ab6a-c4089532f003/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:ae:f4:17,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/73b896bb-7c7d-447e-ab6a-c4089532f003/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
然而誰能解釋這些參數是干什么的？
請仔細閱讀下面兩篇文章
QEMU KVM libvirt 手冊(3) - Storage Media
QEMU KVM Libvirt手冊(8): 半虛擬化設備virtio
machine參數是總線Architecture，通過qemu-system-x86_64 --machine ?查看，default就是參數中的值。
accel=kvm說明虛擬化使用的是kvm
cpu表示處理器的參數以及處理器的一些flags，可以使用命令qemu-system-x86_64 --cpu ?查看
smp是對稱多處理器，
-smp 1,sockets=1,cores=1,threads=1
qemu仿真了一個具有1個vcpu，一個socket，一個core，一個threads的處理器。
socket, core, threads是什么概念呢
（1）socket就是主板上插cpu的槽的數目，也即管理員說的”路“
（2）core就是我們平時說的”核“，即雙核，4核等
（3）thread就是每個core的硬件線程數，即超線程
具體例子，某個服務器是：2路4核超線程（一般默認為2個線程），那么，通過cat /proc/cpuinfo看到的是2*4*2=16個processor，很多人也習慣成為16核了！
SMBIOS全稱System Management BIOS，用於表示x86 architectures的硬件信息，包含BIOS，主板的信息，這里都是openstack，是假的了
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000006.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control
這是一對，用unix socket方式暴露monitor，從而可以通過virsh操作monitor
rtc是指system clock, -no-hpet是指不用更精准的時間。
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 是USB，連接到PCI總線0上，是device 0, function 1
下面兩個是一對
-drive file=/var/lib/nova/instances/73b896bb-7c7d-447e-ab6a-c4089532f003/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
表示硬盤，drive指向文件，device使用virtio，連到pci的總線0上，是device 4, funtion 0
下面兩個是一對
-netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=25
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:ae:f4:17,bus=pci.0,addr=0x3
表示網卡，用tap device，device使用virtio，連接到pci的總線0上，是device 3，function 0
下面兩個是一對
-chardev file,id=charserial0,path=/var/lib/nova/instances/73b896bb-7c7d-447e-ab6a-c4089532f003/console.log
-device isa-serial,chardev=charserial0,id=serial0
是chardev，將log重定向到console.log
下面兩個是一對，是pty
-chardev pty,id=charserial1
-device isa-serial,chardev=charserial1,id=serial1
這是顯卡
-device cirrus-vga,id=video0,bus=pci.0,addr=0x2
這是內存
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
都連接到pci總線上，通過命令virsh # qemu-monitor-command instance-00000024 --hmp "info pci"可以查看pci總線上的所有設備。
這里面有很多半虛擬化的設備，從而可以提高性能
[轉] KVM VirtIO paravirtualized drivers: why they matter
Virtio: An I/O virtualization framework for Linux
QEMU KVM Libvirt手冊(8): 半虛擬化設備virtio
除了硬件的虛擬化和半虛擬化，對於網絡，qemu和kvm本身也有自己的機制
QEMU KVM Libvirt手冊(9): network
同時對於存儲，也有自己的機制
QEMU KVM Libvirt手冊(11): Managing Storage
這一節最后要說的，就是libvirt對虛擬機的管理
有一個強大的工具叫monitor，可以進行多種操作，相當於機器的管理界面，也可以通過virsh進行操作，參考文章QEMU KVM libvirt手冊(2)
最重要的命令行工具就是virsh了，參考QEMU KVM Libvirt手冊(10):Managing Virtual Machines with libvirt
七、Neutron
這一步，就是講instance連接到已經創建好的網絡設備上
步驟33：創建qbr網橋
步驟34：創建veth pair，qvo和qvb
步驟35：將qvb添加到qbr上
步驟36：將qvo添加到br-int上
看起來復雜而清晰的連接過程，為什么要這樣，需要理解neutron中的網絡設備架構
其實很早就有人畫出來了，如下面的圖
在network Node上：
在Compute Node上：
當看到這里，很多人腦袋就大了，openstack為什么要創建這么多的虛擬網卡，他們之間什么關系，這些dl_vlan, mod_vlan_vid都是什么東東啊？
請參考文章neutron的基本原理
neutron的不同的private network之間是隔離的，租戶隔離技術三種常用的VLAN, GRE，VXLAN，各有優缺點
VLAN原理
A virtual LAN (VLAN) is a group of networking devices in the same broadcast domain.
有兩種VLAN
Static VLAN/Port-based VLAN
- manually assign a port on a switch to a VLAN using an Interface Subconfiguration mode command.
Dynamic VLANs
- the switch automatically assigns the port to a VLAN using information from the user device, such as its MAC address, IP address, or even directory information (a user or group name, for instance).
- The switch then consults a policy server, called a VLAN membership policy server (VMPS), which contains a mapping of device information to VLANs.
有兩種connection
Access-Link Connections
- a device that has a standardized Ethernet NIC that understands only standardized Ethernet frames
- Access-link connections can only be associated with a single VLAN.
Trunk Connections
- trunk connections are capable of carrying traffic for multiple VLANs.
IEEE’s 802.1Q
優點
Increased performance
- reducing collisions
- limiting broadcast traffic
- Less need to be routed
Improved manageability
- Manage logical groups
Increased security options
- packets only to other members of the VLAN.
缺點
limited number of VLANs 4000 -> 1000
number of MAC addresses supported in switches
GRE的原理
Generic Routing Encapsulation (GRE) is a tunneling protocol that can encapsulate a wide variety of network layer protocols inside virtual point-to-point links over an Internet Protocol internetwork.
Header
優點
Resolve the VLAN and MAC limitations by encapsulating communications within p2p 'tunnels' which hid the guest MAC information exposing only the MAC addresses of host systems.
L2 to L3, after leaving the encapsulated L2 virtual network, the traffic is forwarded to a gateway which can de-encapsulate traffic and route it out onto the leveraged unencapsulated network.
缺點
Point to point tunnel
Pool extensibility
Few switches can understand GRE Header, so load distribution and ACL (both depends on IPs and ports) can not be applied
VXLAN原理
Allow for virtual machines to live in two disparate networks yet still operate as if they were attached to the same L2.
Components:
- Multicast support, IGMP and PIM
- VXLAN Network Identifier (VNI): 24-bit segment ID
- VXLAN Gateway
- VXLAN Tunnel End Point (VTEP)
- VXLAN Segment/VXLAN Overlay Network
- When VM1 wants to send a packet to VM2, it needs the MAC address of VM2 this is the process that is followed:
- VM1 sends a ARP packet requesting the MAC address associated with 192.168.0.101
- This ARP is encapsulated by VTEP1 into a multicast packet to the multicast group associated with VNI 864
- All VTEPs see the multicast packet and add the association of VTEP1 and VM1 to its VXLAN tables
- VTEP2 receives the multicast packet decapsulates it, and sends the original broadcast on portgroups associated with VNI 864
- VM2 sees the ARP packet and responds with its MAC address
- VTEP2 encapsulates the response as a unicast IP packet and sends it back to VTEP1 using IP routing
- VTEP1 decapsulates the packet and passes it on to VM1
- At this point VM1 knows the MAC address of VM2 and can send directed packets to it as shown in in Figure 2: VM to VM communication:
- VM1 sends the IP packet to VM2 from IP address 192.168.0.100 to 192.168.0.101
- VTEP1 takes the packet and encapsulates it by adding the following headers:
- VXLAN header with VNI=864
- Standard UDP header and sets the UDP checksum to 0×0000, and the destination port being the VXLAN IANA designated port. Cisco N1KV is currently using port ID 8472.
- Standard IP header with the Destination being VTEP2’s IP address and Protocol 0×011 for the UDP packet used for delivery
- Standard MAC header with the MAC address of the next hop. In this case it is the router Interface with MAC address 00:10:11:FE:D8:D2 which will use IP routing to send it to the destination
- VTEP2 receives the packet as it has it’s MAC address as the destination. The packet is decapsulated and found to be a VXLAN packet due to the UDP destination port. At this point the VTEP will look up the associated portgroups for VNI 864 found in the VXLAN header. It will then verify that the target, VM2 in this case, is allowed to receive frames for VNI 864 due to it’s portgroup membership and pass the packet on if the verification passes.
- VM2 receives the packet and deals with it like any other IP packet.
優點
Address 4K VLAN Limitation
Solves mac address scaling issues
Better scalability and failover
缺點
VXLAN expects multicast to be enabled on physical networks, and it does MAC flooding to learn end points.
But IP multicast is usually disabled
Need MAC preprovisioning via a SDN Controller
Software VTEPs may have performance issue
在Openstack中，neutron的很多網絡功能都是由openvswitch實現的，因而本人專門研究了一下openvswitch，參考下面的文章
[轉]Comparing sFlow and NetFlow in a vSwitch
[轉]Rapidly detecting large flows, sFlow vs. NetFlow/IPFIX
對於網絡的管理，有很多好的工具可以使用
HTB Linux queuing discipline manual - user guide筆記
[轉] Firewall and network filtering in libvirt
[轉] XEN, KVM, Libvirt and IPTables
http://tldp.org/HOWTO/Traffic-Control-HOWTO/
http://rlworkman.net/howtos/iptables/iptables-tutorial.html