QEMU KVM Libvirt(12): Live Migration

由於KVM的架構為

Libvirt –> qemu –> KVM

所以對於live migration有兩種方式，一種是qemu + KVM自己的方式，一種是libvirt的方式，當然libvirt也是基於qemu+kvm的方式

qemu + KVM自己的方式為使用monitor

KVM Migration

KVM currently supports savevm/loadvm and offline or live migration Migration commands are given when in qemu-monitor (Alt-Ctrl-2). Upon successful completion, the migrated VM continues to run on the destination host.

Requirements

• The VM image is accessible on both source and destination hosts (located on a shared storage, e.g. using nfs).
• It is recommended an images-directory would be found on the same path on both hosts (for migrations of a copy-on-write image -- an image created on top of a base-image using "qemu-image create -b ...")
• The src and dst hosts must be on the same subnet (keeping guest's network when tap is used).
• Do not use -snapshot qemu command line option.
• For tcp: migration protocol

the guest on the destination must be started the same way it was started on the source.

The live migration process has the following steps:

1. The virtual machine instance is running on the source host.

2. The virtual machine is started on the destination host in the frozen listening mode. The parameters used are the same as on the source host plus the -incoming tcp:ip:port parameter, where ip specifies the IP address and port specifies the port for listening to the incoming migration. If 0 is set as IP address, the virtual machine listens on all interfaces.

3. On the source host, switch to the monitor console and use the migrate -d tcp:destination_ip:port command to initiate the migration.

4. To determine the state of the migration, use the info migrate command in the monitor console on the source host.

5. To cancel the migration, use the migrate_cancel command in the monitor console on the source host.

6. To set the maximum tolerable downtime for migration in seconds, use the migrate_set_downtime number_of_seconds command.

7. To set the maximum speed for migration in bytes per second, use the migrate_set_speed bytes_per_second command.

要進行live migration首先要存儲共享，我們這里用nfs

在一台nfs server上安裝

apt-get install nfs-kernel-server

export如下的文件夾

# cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
#               to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#
/home/cliu8/nfs    *(rw,sync,no_root_squash)

在source機器和destination機器上，都mount這個文件夾

mount 16.158.166.150:/home/cliu8/nfs /home/cliu8/migrate

在文件夾里面放入兩個image

root@escto-bj-hp-z620:/home/cliu8/migrate# ls -l
total 3842908
-rwxr-xr-x 1 root root 1717567488 Jul 18 00:01 ubuntu-14.04.img
-rwxr-xr-x 1 root root 2217869312 Jul 17 22:13 ubuntutest.img

在source機器上啟動

qemu-system-x86_64 -enable-kvm -name ubuntutest  -m 2048 -hda /home/cliu8/migrate/ubuntu-14.04.img -vnc :19 -net nic -net user

在dest機器上同樣啟動，但是多了listen的參數

qemu-system-x86_64 -enable-kvm -name ubuntutest  -m 2048 -hda /home/cliu8/migrate/ubuntu-14.04.img -vnc :19 -net nic -net user -incoming tcp:0:4444

打開source機器的monitor

運行migrate -d tcp:16.158.166.150:4444

這個時候info migrate，顯示Migration status: active

等變成complete，則migration結束

這個時候，另一面的機器已經起來了。

Libvirt Migration

Network data transports

migration的時候的數據傳輸有兩種方式：

Hypervisor native transport

所謂native的transport，就是依賴於hypervisor，也即KVM的自有的機制去做網絡傳輸，不支持加密，可能對於某些hypervisor來講，網絡還需要特殊的配置。

libvirt tunnelled transport

依賴於libvirt的RPC的網絡通路進行網絡傳輸，支持加密。

只要下面的命令能夠執行，就能夠傳輸

virsh -c qemu+ssh://cliu8@16.158.166.150/system list --all
virsh -c qemu+tcp://popsuper1982/system list --all
virsh -c qemu+tls://popsuper1982/system list --all

缺點是有很多額外的性能損耗

Communication control paths/flows

看完了數據通路，我們再來看控制通路

Migration一般涉及三個較色，admin, source, destination

Managed direct migration

admin節點控制整個migration的全過程，admin既控制source, 也控制destination，然而source和destination之間並不交互，因而migration過程中，如果admin掛了，就失敗了。

Managed peer to peer migration

admin節點僅僅和source交互，告訴source，你要migrate到destination，然后source控制整個過程，如果admin在這個過程中掛了，不影響接着進行migration

注意的是，admin登錄source的credential信息和source登錄destination的credential信息是不一樣的。

Unmanaged direct migration

admin和source都不控制migration的過程，而是admin的libvirt直接調用hypervisor的控制器，讓hypervisor自己進行migration。

Configuration file handling

A transient guest only exists while it is running, and has no configuration file stored on disk.

A persistent guest maintains a configuration file on disk even when it is not running.

The virsh command has two flags to influence this behaviour.

The --undefine-source flag will cause the configuration file to be removed on the source host after a successful migration.

The --persist flag will cause a configuration file to be created on the destination host after a successful migration.

我們首先需要使得source和destination之間的libvirt是相互通的。

在source機器上，16.158.166.197

root@escto-bj-hp-z620:/home/cliu8/certtool# ls -l
total 60
-r--r--r-- 1 root root 1204 Jul 17 20:17 certificate_authority_certificate.pem
-r--r--r-- 1 root root 1972 Jul 17 20:17 certificate_authority_key.pem
-r--r--r-- 1 root root   37 Jul 17 20:17 certificate_authority_template.info
-r--r--r-- 1 root root 1379 Jul 17 20:17 escto-bj-hp-z620_client_certificate.pem
-r--r--r-- 1 root root 1968 Jul 17 20:17 escto-bj-hp-z620_client_key.pem
-r--r--r-- 1 root root  139 Jul 17 20:17 escto-bj-hp-z620_client_template.info
-r--r--r-- 1 root root 1310 Jul 17 20:17 escto-bj-hp-z620_server_certificate.pem
-r--r--r-- 1 root root 1968 Jul 17 20:17 escto-bj-hp-z620_server_key.pem
-r--r--r-- 1 root root   91 Jul 17 20:17 escto-bj-hp-z620_server_template.info
-r--r--r-- 1 root root 1371 Jul 17 20:17 popsuper1982_client_certificate.pem
-r--r--r-- 1 root root 1972 Jul 17 20:17 popsuper1982_client_key.pem
-r--r--r-- 1 root root  135 Jul 17 20:17 popsuper1982_client_template.info
-r--r--r-- 1 root root 1306 Jul 17 20:17 popsuper1982_server_certificate.pem
-r--r--r-- 1 root root 1968 Jul 17 20:17 popsuper1982_server_key.pem
-r--r--r-- 1 root root   87 Jul 17 20:17 popsuper1982_server_template.info

root@escto-bj-hp-z620:/home/cliu8/certtool# tree --charset ASCII /etc/pki/
/etc/pki/
|-- CA
|   `-- cacert.pem -> /home/cliu8/certtool/certificate_authority_certificate.pem
|-- libvirt
|   |-- clientcert.pem -> /home/cliu8/certtool/escto-bj-hp-z620_client_certificate.pem
|   |-- private
|   |   |-- clientkey.pem -> /home/cliu8/certtool/escto-bj-hp-z620_client_key.pem
|   |   `-- serverkey.pem -> /home/cliu8/certtool/escto-bj-hp-z620_server_key.pem
|   `-- servercert.pem -> /home/cliu8/certtool/escto-bj-hp-z620_server_certificate.pem
`-- nssdb -> /var/lib/nssdb

配置/etc/libvirt/libvirtd.conf

為了方便測試，我們對tcp, tls不進行密碼設置

listen_tls = 1

listen_tcp = 1

tls_port = "16514"

tcp_port = "16509"

unix_sock_group = "libvirtd"

unix_sock_ro_perms = "0777"

unix_sock_rw_perms = "0770"

auth_unix_ro = "none"

auth_unix_rw = "none"

auth_tcp = "none"

auth_tls = "none"

測試下面的命令都能通過

virsh -c qemu+ssh://cliu8@16.158.166.150/system list --all
virsh -c qemu+tcp://popsuper1982/system list --all
virsh -c qemu+tls://popsuper1982/system list --all

在destination機器上，16.158.166.150

root@popsuper1982:/home/cliu8/certtool# ls -l
total 60
-r--r--r-- 1 root root 1204 Jul 15 22:31 certificate_authority_certificate.pem
-r--r--r-- 1 root root 1972 Jul 15 22:28 certificate_authority_key.pem
-r--r--r-- 1 root root   37 Jul 15 22:26 certificate_authority_template.info
-r--r--r-- 1 root root 1379 Jul 16 00:27 escto-bj-hp-z620_client_certificate.pem
-r--r--r-- 1 root root 1968 Jul 16 00:25 escto-bj-hp-z620_client_key.pem
-r--r--r-- 1 root root  139 Jul 16 00:24 escto-bj-hp-z620_client_template.info
-r--r--r-- 1 root root 1310 Jul 17 20:09 escto-bj-hp-z620_server_certificate.pem
-r--r--r-- 1 root root 1968 Jul 17 20:07 escto-bj-hp-z620_server_key.pem
-r--r--r-- 1 root root   91 Jul 17 20:06 escto-bj-hp-z620_server_template.info
-r--r--r-- 1 root root 1371 Jul 17 20:14 popsuper1982_client_certificate.pem
-r--r--r-- 1 root root 1972 Jul 17 20:13 popsuper1982_client_key.pem
-r--r--r-- 1 root root  135 Jul 17 20:12 popsuper1982_client_template.info
-r--r--r-- 1 root root 1306 Jul 16 00:09 popsuper1982_server_certificate.pem
-r--r--r-- 1 root root 1968 Jul 16 00:06 popsuper1982_server_key.pem
-r--r--r-- 1 root root   87 Jul 16 00:05 popsuper1982_server_template.info
root@popsuper1982:/home/cliu8/certtool# tree --charset ASCII /etc/pki/
/etc/pki/
|-- CA
|   `-- cacert.pem -> /home/cliu8/certtool/certificate_authority_certificate.pem
|-- libvirt
|   |-- clientcert.pem -> /home/cliu8/certtool/popsuper1982_client_certificate.pem
|   |-- private
|   |   |-- clientkey.pem -> /home/cliu8/certtool/popsuper1982_client_key.pem
|   |   `-- serverkey.pem -> /home/cliu8/certtool/popsuper1982_server_key.pem
|   `-- servercert.pem -> /home/cliu8/certtool/popsuper1982_server_certificate.pem
`-- nssdb -> /var/lib/nssdb

virsh -c qemu+ssh://cliu8@16.158.166.197/system list --all
virsh -c qemu+tcp://escto-bj-hp-z620/system list --all
virsh -c qemu+tls://escto-bj-hp-z620/system list –all

在source機器上，啟動一個虛擬機virsh start ubuntu-14.04

# virsh dumpxml ubuntu-14.04
<domain type='kvm' id='55'>
  <name>ubuntu-14.04</name>
  <uuid>0f0806ab-531d-6134-5def-c5b495529284</uuid>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/home/cliu8/migrate/ubuntu-14.04.img'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:11:9b:d5:11'/>
      <source bridge='ubuntu_br'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='18a45d7e-d96b-4b9e-9d92-dc9ff3ea77e0'/>
      </virtualport>
      <target dev='vnet8'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/16'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/16'>
      <source path='/dev/pts/16'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='5908' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none'/>
</domain>

其中cdrom和usb的都應該去掉，才能migration成功。

# virsh migrate --verbose --live --persistent ubuntu-14.04 qemu+tcp://popsuper1982/system     
Migration: [100 %]