目錄
文章目錄
前文列表
NOTE：本文語境限於 OpenStack 原生 Libvirt Driver（QEMU-KVM Hypervisor）。
虛擬機遷移的應用場景
- 當某個計算節點發生故障修復時，將其之上的虛擬機遷移出去。
- 對某個計算節點進行硬件升級維護時，將其之上的虛擬機遷移出去。
- 當某個計算節點的負載超出安全閾值時，將其之上的一些虛擬機遷移出去。
- 對某個 REGION 或 AZ 中的計算節點進行負載均衡。
需要遷移的虛擬機數據類型
根據虛擬機啟動方式、虛擬機磁盤類型的不同可以分為：
-
Boot from image
- 本地系統盤文件：Root file
- 本地數據盤文件：Ephemeral file、Swap file
- 本地配置文件：console.log、disk.info
- 共享數據盤：Volumes
- 內存數據：RAM
- 其他：Config Drive
-
Boot from volume
- 共享系統盤：Root Volume
- 本地數據盤文件：Ephemeral file、Swap file
- 本地配置文件：console.log、disk.info
- 共享數據盤：Volumes
- 虛擬機的內存數據：RAM
- 其他：Config Drive
NOTE：更多 OpenStack 虛擬機磁盤文件類型與存儲方式請瀏覽《OpenStack 虛擬機的磁盤文件類型與存儲方式》。
虛擬機遷移的存儲場景
文件存儲
典型的文件存儲是 NAS（Network Attached Storage，網絡接入存儲）、NFS（Network File System，網絡文件系統）。多個計算節點可以通過 NFS 共享 instances_path 存放磁盤文件，Cinder 也支持 NFS Backend 將 Volumes 存放到共享目錄下。可見，在共享存儲的場景中，需要遷移的僅僅是虛擬機的內存數據。
塊存儲
典型的塊存儲（Block Storage）是 SAN（Storage Area Network，區域存儲網絡）。塊存儲通過 iSCSI、FC 等協議將塊設備 Attach 到應用服務器上，並被文件系統層接管，數據以塊的形式存儲在 Volume 里。同樣的，Nova、Glance、Cinder 都可以接入 Block Storage Backend。即虛擬機使用了共享的系統盤、數據盤，Local 只存放 Ephemeral file、Swap file、console.log、disk.info 等磁盤及配置文件。如果虛擬機沒有使用 Ephemeral、Swap Disks 的話，那么需要遷移的也同樣只有內存數據。
從某種層面上看，我們不妨將文件存儲、塊存儲歸納為共享存儲行列。但需要注意的是，嚴格來說純粹的塊存儲並不能稱之為共享存儲，因為塊設備無法做到在多個掛載點上同時刷寫數據，這需要業務層的支持，例如 CInder Multi-Attach。這也導致了在遷移場景中，對文件存儲、塊存儲的處理方式並不相同：使用塊存儲的遷移過程，塊設備需要經歷先 Detach、再重定向 Attach 的過程；而使用文件存儲的遷移過程中只需要在目的節點上直接訪問 mountd 目錄即可。
值得一提的是，NAS 和 SAN 作為傳統存儲解決方案分別提供了文件存儲和塊存儲。而 Ceph 作為統一存儲解決方案，能夠同時提供文件存儲、塊存儲和對象存儲，現在已經被大量使用到生產環境中。
非共享存儲
非共享存儲場景中，虛擬機的磁盤文件存放方式為 Local，這就需要對虛擬機的內存數據、本地磁盤文件均進行遷移。遷移方式就是數據塊級別的拷貝，簡稱塊遷移。顯然，這種場景對熱遷移並不友好，因為拷貝時間太長會提高數據的丟失率（e.g. 拷貝過程中的網絡問題）。
遷移的類型
為人熟知的遷移類型有冷遷移和熱遷移，這兩個概念很好區別，以遷移過程中是否需要關閉業務主機作為辨識。
-
冷遷移：即關閉虛擬機、數據遷移。需要遷移的只有系統盤數據、數據盤數據，而無需遷移內存數據，使用塊遷移方式。
- 好處：操作簡單，遷移方式靈活，虛擬機不產生動態數據，所以數據丟失率小。
- 缺點：運行在虛擬機之上的業務被中斷。
-
熱遷移：又稱動態遷移、在線遷移，是一種用戶無感的遷移方式。虛擬機不需要關機，業務不被中斷，但相對的是一種復雜的遷移方式。
- 好處：用戶無感，平滑遷移，有效保證業務高可用性。
- 缺點：操作復雜，數據丟失率大。
Non-live migration, also known as cold migration or simply migration. The instance is shut down, then moved to another hypervisor and restarted. The instance recognizes that it was rebooted, and the application running on the instance is disrupted.
Live migration, The instance keeps running throughout the migration. This is useful when it is not possible or desirable to stop the application running on the instance. Live migrations can be classified further by the way they treat instance storage:
- Shared storage-based live migration. The instance has ephemeral disks that are located on storage shared between the source and destination hosts.
- Block live migration, or simply block migration. The instance has ephemeral disks that are not shared between the source and destination hosts. Block migration is incompatible with read-only devices such as CD-ROMs and Configuration Drive (config_drive).
- Volume-backed live migration. Instances use volumes rather than ephemeral disks.
遷移的方式
根據虛擬機數據類型、存儲場景的不同，遷移方式可以分為：
-
冷遷移
- 本地磁盤文件 — 基於 scp 指令的塊遷移
- 塊存儲卷 — 重定向掛載
- 文件存儲目錄 — 直接訪問本地文件系統
-
熱遷移
- 本地磁盤文件 — 基於 scp 指令的塊遷移，或基於 Hypervisor 的塊遷移（視乎 Hypervisor 是否對除內存數據之外的數據進行塊遷移）
- 塊存儲卷 — 重定向掛載
- 文件存儲目錄 — 直接訪問本地文件系統
- 內存數據 — 基於 Hypervisor 的塊遷移
執行虛擬機冷遷移
遷移場景信息：
- Boot from image
- Nova 沒有接入后端存儲，虛擬機磁盤文件存放方式為 Local
Step 1. 保證源計算節點和目的計算節點的 nova 用戶可以進行 SSH 免密登錄，因為 nova-compute.service 默認是由 nova 用戶啟動的，該服務進程會在源計算節點和目的計算節點之間使用 scp 指令進行數據拷貝。否則就會出現如下異常：
2019-03-15 03:33:21.428 10639 ERROR oslo_messaging.rpc.server ResizeError: Resize error: not able to execute ssh command: Unexpected error while running command.
2019-03-15 03:33:21.428 10639 ERROR oslo_messaging.rpc.server Command: ssh -o BatchMode=yes 172.17.1.16 mkdir -p /var/lib/nova/instances/1365380a-a532-4811-8784-57f507acac46
2019-03-15 03:33:21.428 10639 ERROR oslo_messaging.rpc.server Exit code: 255
2019-03-15 03:33:21.428 10639 ERROR oslo_messaging.rpc.server Stdout: u''
2019-03-15 03:33:21.428 10639 ERROR oslo_messaging.rpc.server Stderr: u'Permission denied (publickey,gssapi-keyex,gssapi-with-mic).\r\n'
Step 2. 關閉 SELinux 或者執行下述指令讓 SSH 免密登錄認證文件可被訪問
chcon -R -t ssh_home_t /var/lib/nova/.ssh/authorized_keys
NOTE：SELinux 相關日志 /var/log/audit/audit.log
Step 3. 執行遷移
[stack@undercloud (overcloudrc) ~]$ openstack server create --image c9debff2-cd87-4688-b712-87a2948461ce --flavor a0dd32df-8c1b-47ed-9b7c-88612a5dd78d --nic net-id=11d8d379-dcd9-46ff-9cd1-25d2737affb4 tst-block-migrate-vm
+--------------------------------------+-----------------------------------------------+
| Field | Value |
+--------------------------------------+-----------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| adminPass | WswHKEmcPnV3 |
| config_drive | |
| created | 2019-03-15T09:32:17Z |
| flavor | 1U2G (a0dd32df-8c1b-47ed-9b7c-88612a5dd78d) |
| hostId | |
| id | 80996760-0c30-4e2a-847a-b9d882182df5 |
| image | cirros (c9debff2-cd87-4688-b712-87a2948461ce) |
| key_name | None |
| name | tst-block-migrate-vm |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| project_id | a6c78435075246f3aa5ab946b87086c5 |
| properties | |
| security_groups | [{u'name': u'default'}] |
| status | BUILD |
| updated | 2019-03-15T09:32:17Z |
| user_id | 4fe574569664493bbd660abfe762a630 |
+--------------------------------------+-----------------------------------------------+
[stack@undercloud (overcloudrc) ~]$ openstack server show tst-block-migrate-vm
+--------------------------------------+----------------------------------------------------------+
| Field | Value |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | ovs |
| OS-EXT-SRV-ATTR:host | overcloud-ovscompute-1.localdomain |
| OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-ovscompute-1.localdomain |
| OS-EXT-SRV-ATTR:instance_name | instance-0000008e |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2019-03-15T09:32:31.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | net1=10.0.1.14 |
| config_drive | |
| created | 2019-03-15T09:32:17Z |
| flavor | 1U2G (a0dd32df-8c1b-47ed-9b7c-88612a5dd78d) |
| hostId | 9f1230901ddf3fe0e1a41e1c650a784c122b791f89fdf66a40cff3d6 |
| id | 80996760-0c30-4e2a-847a-b9d882182df5 |
| image | cirros (c9debff2-cd87-4688-b712-87a2948461ce) |
| key_name | None |
| name | tst-block-migrate-vm |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| project_id | a6c78435075246f3aa5ab946b87086c5 |
| properties | |
| security_groups | [{u'name': u'default'}] |
| status | ACTIVE |
| updated | 2019-03-15T09:32:32Z |
| user_id | 4fe574569664493bbd660abfe762a630 |
+--------------------------------------+----------------------------------------------------------+
[stack@undercloud (overcloudrc) ~]$ openstack server migrate --block-migration --wait tst-block-migrate-vm
Complete
[stack@undercloud (overcloudrc) ~]$ openstack server show tst-block-migrate-vm
+--------------------------------------+----------------------------------------------------------+
| Field | Value |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | ovs |
| OS-EXT-SRV-ATTR:host | overcloud-ovscompute-0.localdomain |
| OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-ovscompute-0.localdomain |
| OS-EXT-SRV-ATTR:instance_name | instance-0000008e |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2019-03-15T09:33:52.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | net1=10.0.1.14 |
| config_drive | |
| created | 2019-03-15T09:32:17Z |
| flavor | 1U2G (a0dd32df-8c1b-47ed-9b7c-88612a5dd78d) |
| hostId | 0f2ec590cd73fe0e9522f1ba715dae7a7d4b884e15aa8254defe85d0 |
| id | 80996760-0c30-4e2a-847a-b9d882182df5 |
| image | cirros (c9debff2-cd87-4688-b712-87a2948461ce) |
| key_name | None |
| name | tst-block-migrate-vm |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| project_id | a6c78435075246f3aa5ab946b87086c5 |
| properties | |
| security_groups | [{u'name': u'default'}] |
| status | ACTIVE |
| updated | 2019-03-15T09:34:53Z |
| user_id | 4fe574569664493bbd660abfe762a630 |
+--------------------------------------+----------------------------------------------------------+
NOTE：上述執行遷移時沒有顯式選中目的計算節點，交由 nova-scheduler.service 負責調度。
冷遷移日志分析
從上述遷移場景信息可知，Nova 會對該虛擬機使用塊遷移方式進行磁盤文件的拷貝，這一點也體現在了操作日志中。
源主機日志分析：
# 開始進入遷移邏輯。
Starting migrate_disk_and_power_off
# 在目的計算節點上嘗試創建 tmp 文件，以此來判斷源計算節點和目的計算節點是否使用了共享存儲
Creating file /var/lib/nova/instances/80996760-0c30-4e2a-847a-b9d882182df5/8ac1bb9977bc4b4b948c4c8fdad9f1f6.tmp on remote host 172.17.1.16 create_file
# 創建 tmp 文件失敗，表示沒有使用共享存儲，因為虛擬機目錄不存在
'ssh -o BatchMode=yes 172.17.1.16 touch /var/lib/nova/instances/80996760-0c30-4e2a-847a-b9d882182df5/8ac1bb9977bc4b4b948c4c8fdad9f1f6.tmp' failed. Not Retrying.
# 在目的計算節點上創建虛擬機目錄
Creating directory /var/lib/nova/instances/80996760-0c30-4e2a-847a-b9d882182df5 on remote host 172.17.1.16
# 關閉虛擬機電源
Shutting down instance
Instance shutdown successfully after 35 seconds.
# Hypervisor 層刪除虛擬機實例
Instance destroyed successfully.
# 重命名虛擬機目錄
mv /var/lib/nova/instances/80996760-0c30-4e2a-847a-b9d882182df5 /var/lib/nova/instances/80996760-0c30-4e2a-847a-b9d882182df5_resize
# 拷貝虛擬機磁盤文件及配置文件至目的主機
scp -r /var/lib/nova/instances/80996760-0c30-4e2a-847a-b9d882182df5_resize/disk 172.17.1.16:/var/lib/nova/instances/80996760-0c30-4e2a-847a-b9d882182df5/disk
scp -r /var/lib/nova/instances/80996760-0c30-4e2a-847a-b9d882182df5_resize/disk.info 172.17.1.16:/var/lib/nova/instances/80996760-0c30-4e2a-847a-b9d882182df5/disk.info
# Nova 層面虛擬機已停止
VM Stopped (Lifecycle Event)
# 正式刪除源主機虛擬機目錄
rm -rf /var/lib/nova/instances/80996760-0c30-4e2a-847a-b9d882182df5_resize
# 移除虛擬機網絡設備
Unplugging vif VIFBridge(active=True,address=fa:16:3e:d0:f6:a4,bridge_name='qbr15c7b577-89',has_traffic_filtering=True,id=15c7b577-89f5-46f6-8111-5f4e0c8ebaa1,network=Network(11d8d379-dcd9-46ff-9cd1-25d2737affb4),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap15c7b577-89')
brctl delif qbr15c7b577-89 qvb15c7b577-89
ip link set qbr15c7b577-89 down
brctl delbr qbr15c7b577-89
ovs-vsctl --timeout=120 -- --if-exists del-port br-int qvo15c7b577-89
ip link delete qvo15c7b577-89
目的主機日志分析：
# Nova 層面新建虛擬機，預扣虛擬機資源
Claim successful
# 遷移中
Migrating
# 更新虛擬機 vNIC 的 Port binding:host_id 信息
Updating port 15c7b577-89f5-46f6-8111-5f4e0c8ebaa1 with attributes {'binding:host_id': u'overcloud-ovscompute-0.localdomain'}
# 創建虛擬機鏡像
Creating image
# 檢查是否可以 resize 虛擬機磁盤文件
Checking if we can resize image /var/lib/nova/instances/80996760-0c30-4e2a-847a-b9d882182df5/disk.
Cannot resize image /var/lib/nova/instances/80996760-0c30-4e2a-847a-b9d882182df5/disk
# 確定虛擬機 console.log 日志文件存在
Ensure instance console log exists: /var/lib/nova/instances/80996760-0c30-4e2a-847a-b9d882182df5/console.log
# 組裝 GuestOS 的 XML 文件
End _get_guest_xml
# 添加虛擬機網絡虛擬設備
Plugging vif VIFBridge(active=False,address=fa:16:3e:d0:f6:a4,bridge_name='qbr15c7b577-89',has_traffic_filtering=True,id=15c7b577-89f5-46f6-8111-5f4e0c8ebaa1,network=Network(11d8d379-dcd9-46ff-9cd1-25d2737affb4),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap15c7b577-89')
brctl addbr qbr15c7b577-89
brctl setfd qbr15c7b577-89 0
brctl stp qbr15c7b577-89 off
brctl setageing qbr15c7b577-89 0
tee /sys/class/net/qbr15c7b577-89/bridge/multicast_snooping
tee /proc/sys/net/ipv6/conf/qbr15c7b577-89/disable_ipv6
ip link add qvb15c7b577-89 type veth peer name qvo15c7b577-89
ip link set qvb15c7b577-89 up
ip link set qvb15c7b577-89 promisc on
ip link set qvb15c7b577-89 mtu 1450
ip link set qvo15c7b577-89 up
ip link set qvo15c7b577-89 promisc on
ip link set qvo15c7b577-89 mtu 1450
ip link set qbr15c7b577-89 up
brctl addif qbr15c7b577-89 qvb15c7b577-89
ovs-vsctl -- --may-exist add-br br-int -- set Bridge br-int datapath_type=system
ovs-vsctl --timeout=120 -- --if-exists del-port qvo15c7b577-89 -- add-port br-int qvo15c7b577-89 -- set Interface qvo15c7b577-89 external-ids:iface-id=15c7b577-89f5-46f6-8111-5f4e0c8ebaa1 external-ids:iface-status=active external-ids:attached-mac=fa:16:3e:d0:f6:a4 external-ids:vm-uuid=80996760-0c30-4e2a-847a-b9d882182df
ip link set qvo15c7b577-89 mtu 1450
# Noca 層面虛擬機已啟動
Instance running successfully.
VM Started (Lifecycle Event)
NOTE：雖然上述將源主機和目的主機的操作日志分開記錄，但實際上兩者的 nova-compute.service 是交叉交互的，並非源主機的遷移操作處理完了之后再開始進行目的主機的遷移操作處理。
執行虛擬機熱遷移
遷移場景信息：
- Boot from image
- Nova 沒有接入后端存儲，虛擬機磁盤文件存放方式為 Local
- Cinder 接入 LVM Backend
- 具有多個塊設備
- 具有多個端口
- 具有 NUMA 親和
- 具有 CPU 綁定
從遷移場景信息可知，虛擬機的 Local 磁盤文件、內存數據都通過 Libvirt Live Migration 完成遷移；共享塊設備通過重定向掛載完成遷移；端口設備通過 Neutron 完成虛擬網絡設備的創建和刪除。
為了保障 OpenStack 虛擬機熱遷移的正常運行，需要滿足幾個前提條件：
- 源和目標節點的 CPU 類型要一致。
- 源和目標節點的 Libvirt 版本要一致。
- 源和目標節點能相互識別對方的 hostname（DNS）。
- 虛擬機可能會使用 Config Driver 保存 metadata 或 user data。在 Block Migration 模式下，Config Driver 也會被遷移到目標節點。由於目前 Libvirt 只支持遷移 vfat 類型的 Config Driver，所以需要在 nova.conf 中明確指明啟動虛擬機時創建 vfat 類型的 Config Driver，以保持一致。
配置 Libvirt 使用 SSH 協議進行數據傳輸：
[libvirt]
...
live_migration_uri=qemu+ssh://nova_migration@%s/system?keyfile=/etc/nova/migration/identity
qemu+ssh：使用 ssh 協議nova_migration：執行 ssh 的用戶%s：計算節點 hostname，e.g.nova_migration@cpu01keyfile：安全通信的 ssh 私鑰
除此之外還可以使用 TCP 協議進行數據傳輸：
live_migration_uri=qemu+tcp://nova_migration@%s/system
如果使用了 TCP 協議，那么還得將源和目標節點的 Libvirt TCP 遠程監聽服務打開：
# /etc/libvirt/libvirtd.conf
listen_tls = 0
listen_tcp = 1
auth_tcp = "none"
# /etc/init/libvirt-bin.conf
# options passed to libvirtd, add "-l" to listen on tcp
env libvirtd_opts="-d -l"
# /etc/default/libvirt-bin
libvirtd_opts="-d -l"
NOTE：在熱遷移時也可能會采用塊遷移，但這不是值得推薦的方式。
Block live migration requires copying disks from the source to the destination host. It takes more time and puts more load on the network. Shared-storage and volume-backed live migration does not copy disks.
[stack@undercloud (overcloudrc) ~]$ openstack server show VM1
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | ovs |
| OS-EXT-SRV-ATTR:host | overcloud-ovscompute-0.localdomain |
| OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-ovscompute-0.localdomain |
| OS-EXT-SRV-ATTR:instance_name | instance-000000a0 |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2019-03-19T08:04:50.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | net1=10.0.1.17, 10.0.1.8, 10.0.1.16, 10.0.1.10, 10.0.1.18, 10.0.1.19 |
| config_drive | |
| created | 2019-03-19T08:04:04Z |
| flavor | Flavor1 (2ff09ec5-19e4-40b9-a52e-6026652c0788) |
| hostId | 0f2ec590cd73fe0e9522f1ba715dae7a7d4b884e15aa8254defe85d0 |
| id | a2855dfd-c6e5-4cbf-9fdf-4b083cc8ec37 |
| image | CentOS-7-x86_64-GenericCloud (0aff2888-47f8-4133-928a-9c54414b3afb) |
| key_name | stack |
| name | VM1 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| project_id | a6c78435075246f3aa5ab946b87086c5 |
| properties | |
| security_groups | [{u'name': u'default'}, {u'name': u'default'}, {u'name': u'default'}, {u'name': u'default'}, {u'name': u'default'}, {u'name': u'default'}] |
| status | ACTIVE |
| updated | 2019-03-19T08:04:50Z |
| user_id | 4fe574569664493bbd660abfe762a630 |
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+
[stack@undercloud (overcloudrc) ~]$ openstack server add volume VM1 volume1
[stack@undercloud (overcloudrc) ~]$ openstack server add volume VM1 volume2
[stack@undercloud (overcloudrc) ~]$ openstack server add volume VM1 volume3
[stack@undercloud (overcloudrc) ~]$ openstack server add volume VM1 volume4
[stack@undercloud (overcloudrc) ~]$ openstack server add volume VM1 volume5
[stack@undercloud (overcloudrc) ~]$ openstack server migrate --block-migration --live overcloud-ovscompute-1.localdomain --wait VM1
Complete
[stack@undercloud (overcloudrc) ~]$ openstack server show VM1
+--------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | ovs |
| OS-EXT-SRV-ATTR:host | overcloud-ovscompute-1.localdomain |
| OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-ovscompute-1.localdomain |
| OS-EXT-SRV-ATTR:instance_name | instance-000000a0 |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2019-03-19T08:04:50.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | net1=10.0.1.17, 10.0.1.8, 10.0.1.16, 10.0.1.10, 10.0.1.18, 10.0.1.19 |
| config_drive | |
| created | 2019-03-19T08:04:04Z |
| flavor | Flavor1 (2ff09ec5-19e4-40b9-a52e-6026652c0788) |
| hostId | 9f1230901ddf3fe0e1a41e1c650a784c122b791f89fdf66a40cff3d6 |
| id | a2855dfd-c6e5-4cbf-9fdf-4b083cc8ec37 |
| image | CentOS-7-x86_64-GenericCloud (0aff2888-47f8-4133-928a-9c54414b3afb) |
| key_name | stack |
| name | VM1 |
| os-extended-volumes:volumes_attached | [{u'id': u'afbe0783-50b8-4036-b59a-69b94dbdb630'}, {u'id': u'27fc8950-6e98-4ba7-9366-907e8fd2a90a'}, {u'id': u'df8b33a8-6d8c-4e0e-a742-869fec4ff923'}, {u'id': u'534bb675-4d8c-4380-8bd2-4aeaedbcda40'}, {u'id': u'623a513a-2cca- |
| | 47e5-9426-71a154cbe0c0'}] |
| progress | 0 |
| project_id | a6c78435075246f3aa5ab946b87086c5 |
| properties | |
| security_groups | [{u'name': u'default'}, {u'name': u'default'}, {u'name': u'default'}, {u'name': u'default'}, {u'name': u'default'}, {u'name': u'default'}] |
| status | ACTIVE |
| updated | 2019-03-19T08:18:39Z |
| user_id | 4fe574569664493bbd660abfe762a630 |
+--------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
NUMA 親和和 CPU 綁定遷移情況：
# 源主機
[root@overcloud-ovscompute-0 nova]# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8
node 0 size: 4095 MB
node 0 free: 1273 MB
node 1 cpus: 9 10 11 12 13 14 15
node 1 size: 4096 MB
node 1 free: 2410 MB
node distances:
node 0 1
0: 10 20
1: 20 10
[root@overcloud-ovscompute-0 nova]# virsh list
Id Name State
----------------------------------------------------
11 instance-000000a0 running
[root@overcloud-ovscompute-0 nova]# virsh vcpuinfo instance-000000a0
VCPU: 0
CPU: 0
State: running
CPU time: 50.1s
CPU Affinity: y---------------
VCPU: 1
CPU: 1
State: running
CPU time: 26.8s
CPU Affinity: -y--------------
[root@overcloud-ovscompute-0 nova]# virsh vcpupin instance-000000a0
VCPU: CPU Affinity
----------------------------------
0: 0
1: 1
# 目的主機
[root@overcloud-ovscompute-1 nova]# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8
node 0 size: 4095 MB
node 0 free: 1420 MB
node 1 cpus: 9 10 11 12 13 14 15
node 1 size: 4096 MB
node 1 free: 2270 MB
node distances:
node 0 1
0: 10 20
1: 20 10
[root@overcloud-ovscompute-1 nova]# virsh vcpuinfo instance-000000a0
VCPU: 0
CPU: 0
State: running
CPU time: 2.9s
CPU Affinity: y---------------
VCPU: 1
CPU: 1
State: running
CPU time: 1.2s
CPU Affinity: -y--------------
[root@overcloud-ovscompute-1 nova]# virsh vcpupin instance-000000a0
VCPU: CPU Affinity
----------------------------------
0: 0
1: 1
熱遷移日志分析
源主機日志分析：
# 通過創建 tmpfile 來檢測是否使用了共存存儲
Check if temp file /var/lib/nova/instances/tmpZ0Bj8s exists to indicate shared storage is being used for migration. Exists? False
# 開始熱遷移
Starting monitoring of live migration _live_migration /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6566
# 輪詢監控 libvirtd 熱遷移狀態並打印遷移進度日志，並動態傳遞 downtime（最大停機時間）
# 因為虛擬機的數據仍會不斷變化，所以最終遷移的 Size 往往會大於 data_gb
Current None elapsed 0 steps [(0, 46), (300, 47), (600, 48), (900, 51), (1200, 57), (1500, 66), (1800, 84), (2100, 117), (2400, 179), (2700, 291), (3000, 500)] update_downtime /usr/lib/python2.7/site-packages/nova/virt/libvirt/migration.py:348
Increasing downtime to 46 ms after 0 sec elapsed time
Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0)
# Nova 層面暫停虛擬機
VM Paused (Lifecycle Event)
# 虛擬機數據遷移完成
Migration operation has completed
Migration operation thread has finished
# 卸載虛擬機的共享塊設備
calling os-brick to detach iSCSI Volume disconnect_volume
iscsiadm -m node -T iqn.2010-10.org.openstack:volume-afbe0783-50b8-4036-b59a-69b94dbdb630 -p 172.17.3.18:3260 --op delete
Checking to see if SCSI volumes sdc have been removed.
SCSI volumes sdc have been removed.
# Nova 層面虛擬機已停止
VM Stopped (Lifecycle Event)
# 拔出虛擬機網絡
Unplugging vif VIFBridge
# 刪除虛擬機本地磁盤文件
mv /var/lib/nova/instances/a2855dfd-c6e5-4cbf-9fdf-4b083cc8ec37 /var/lib/nova/instances/a2855dfd-c6e5-4cbf-9fdf-4b083cc8ec37_del
Deleting instance files /var/lib/nova/instances/a2855dfd-c6e5-4cbf-9fdf-4b083cc8ec37_del
Deletion of /var/lib/nova/instances/a2855dfd-c6e5-4cbf-9fdf-4b083cc8ec37_del complete
# 遷移完成
Migrating instance to overcloud-ovscompute-1.localdomain finished successfully.
Live migration monitoring is all done
目的主機日志分析：
# 創建 tmpfile 檢測是否使用共享存儲
Creating tmpfile /var/lib/nova/instances/tmpZ0Bj8s to notify to other compute nodes that they should mount the same storage.
# 創建虛擬機本地磁盤文件
Creating instance directory: /var/lib/nova/instances/a2855dfd-c6e5-4cbf-9fdf-4b083cc8ec37
touch -c /var/lib/nova/instances/_base/ff34147b1062cd454ae2a8959f069e2e18691ec9
qemu-img create -f qcow2 -o backing_file=/var/lib/nova/instances/_base/ff34147b1062cd454ae2a8959f069e2e18691ec9 /var/lib/nova/instances/a2855dfd-c6e5-4cbf-9fdf-4b083cc8ec37/disk
Creating disk.info with the contents: {u'/var/lib/nova/instances/a2855dfd-c6e5-4cbf-9fdf-4b083cc8ec37/disk': u'qcow2'}
Checking to make sure images and backing files are present before live migration.
# 檢查磁盤文件是否可以 Resize
Checking if we can resize image /var/lib/nova/instances/a2855dfd-c6e5-4cbf-9fdf-4b083cc8ec37/disk. size=10737418240
qemu-img resize /var/lib/nova/instances/a2855dfd-c6e5-4cbf-9fdf-4b083cc8ec37/disk 10737418240
# 掛載虛擬機的共享塊設備
Connecting volumes before live migration.
Calling os-brick to attach iSCSI Volume connect_volume /usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py:63
Trying to connect to iSCSI portal 172.17.3.18:3260
iscsiadm -m node -T iqn.2010-10.org.openstack:volume-afbe0783-50b8-4036-b59a-69b94dbdb630 -p 172.17.3.18:3260
Attached iSCSI volume {'path': u'/dev/sda', 'scsi_wwn': '360014052de14ef00f124a939740ba645', 'type': 'block'}
# 插入虛擬機網絡
Plugging VIFs before live migration.
# 更新 Port 信息
Port 35f7ede8-2a78-44b6-8c65-108e6f1080aa updated with migration profile {'migrating_to': 'overcloud-ovscompute-1.localdomain'} successfully
# Nova 層面虛擬機已啟動
VM Started (Lifecycle Event)
VM Resumed (Lifecycle Event)
從上述日志可以看出，熱遷移的關鍵動作是交由 Hypervisor 層完成的，Nova 只是對 Hypervisor Live Migration 功能進行了封裝和調度管理。在上例中 Libvirt Live Migration 通過 SSH 協議將虛擬機的本地磁盤文件、內存數據一並傳輸到目的主機。
參考資料
https://developers.redhat.com/blog/2015/03/24/live-migrating-qemu-kvm-virtual-machines/
https://www.cnblogs.com/sammyliu/p/4572287.html
https://docs.openstack.org/nova/pike/admin/configuring-migrations.html
https://docs.openstack.org/nova/pike/admin/live-migration-usage.html
https://blog.csdn.net/lemontree1945/article/details/79901874
https://www.ibm.com/developerworks/cn/linux/l-cn-mgrtvm1/index.html
https://blog.csdn.net/hawkerou/article/details/53482268