OpenStack中的ImageCacheManager管理計算節點鏡像緩存文件

假想過程

1. compute節點的driver檢查本地鏡像存儲路徑，沒有找到鏡像文件，於是從glance api利用image.fetch()下載鏡像到本地。

2. hypervisor層使用鏡像：從driver層拿到鏡像路徑后，hypervisor以自己的方式使用鏡像，比如import等操作。

3. 使用完成后，hypervisor中已有了該鏡像文件，調用返回給driver繼續處理。

4. driver可以根據自身需要，比如為了節省空間，該鏡像文件不再存儲，刪除之。

相關conf選項 

在nova.conf中，

remove_unused_base_images = True  # 允許刪除不再使用的、舊的image文件，以節省計算節點硬盤空間
remove_unused_original_minimum_age_seconds = 86400  # 如無instance使用，則原image文件在1天之后會被刪除
remove_unused_resized_minimum_age_seconds = 3600  # 如無instance使用，resize過的image文件（cow格式）在1小時之后會被刪除
image_cache_subdirectory_name = _base  # 鏡像文件的存儲路徑，該目錄是相對於state_path的路徑名，libvirt上最終為：

# /var/lib/nova/instances/_base/
base_dir = os.path.join(CONF.instances_path,
                        CONF.image_cache_subdirectory_name)

image_cache_manager_interval = 2400  # 定時任務，image cache管理器的運行頻率

ImageCacheManager執行過程

1. 在compute.manager中以周期任務的方式運行，默認頻率是image_cache_manager_interval=40分鍾。

該配置值按理來說不應超過24小時（但doc中並沒有給出該提示），原因在3說明。

2. 檢查self.driver是否支持"has_imagecache"能力，libvirt默認支持，有的driver是不支持的。

3. storage_users.py中的兩個函數register_xx和get_xx用於管理/var/lib/nova/instances/compute_nodes文件，

文件內容為dict，key為host名，value為該host更新compute_nodes文件的時間戳。由本步驟依次執行register和get可知，

該文件用於維護哪些計算節點正在使用本路徑作為鏡像存儲路徑，這些計算節點會周期性更新該文件中的時間戳，

表示：我還有可能在使用該路徑中的鏡像，請不要刪除其中的文件！get_xx函數會判斷時間戳是否在24小時之內，

並且只返回24小時之內有更新的host名稱列表。

4. manager->driver->imagecache.update()，真正開始檢查該目錄下的鏡像文件，並且執行刪除無用的舊image的動作。

(1) _reset_state()清空上次檢查的所有結果

(2) _scan_base_images()遍歷目錄下的鏡像文件，根據文件名格式判斷是image還是resized image還是swap文件，會分別放入不同列表中

(3) _list_running_instances()遍歷all_instances，檢查image/bdms等信息，返回used_images和used_swap_images，

含有local/remote信息（依據第3步的host名稱列表區分）

(4) _age_and_verify_cached_images()對上一步返回的used_images信息做sha1，比對scan得到的當前目錄中的images文件，

確定哪些是正在使用的文件（會調用os.utime(path)更新文件的修改時間），哪些是足夠舊的不再使用的文件，最終會刪除它們。

為不支持image cache manage的driver編寫imagecache，要注意鎖的使用（TODO）

參考libvirt.imagecache的編寫思路，在_remove_old_enough_file(self, base_file, maxage, remove_lock=True)時會加鎖和刪除鎖文件，

使用中有兩種情形：_remove_swap_file()-->remove_lock=False，_remove_base_file()-->remove_lock=True，即base文件刪除時，

會刪除對應的鎖文件，swap文件刪除時，不會刪除該鎖文件（這又是為什么？）。

 

執行os.remove時，會獲取一個鎖文件，鎖路徑為self.lock_path = os.path.join(CONF.instances_path, 'locks')：nova/instances/locks/{sha1 of image_id}

鎖的目的：確定用戶目前不正在使用該鏡像文件？在獲取鎖后會二次檢查是否old：user of the file might have come along while we were waiting for the lock.

 

此時還有另外一個長期存在的鎖文件：nova/instances/locks/nova-{sha1 of image_id}，comment顯示：

The lock file will be constructed first time the image file was accessed.

該鎖文件就是由_remove_old_enough_file()的remove_lock標志控制是否刪除的，即如上所述，base文件刪除時，會刪除對應的鎖文件，

swap文件刪除時，不會刪除該鎖文件（這又是為什么？）。

檢查鎖文件的存放目錄發現，該鎖文件從創建后就一直存在。

 

由於對OpenStack鎖機制的不了解，目前無法確定這兩種鎖在image cache manage中的用法和作用。能想到的競態條件：如果存放路徑是

一個共享目錄的話，那么多個imagecahcemanager可能會對同一個image進行刪除操作（這種情況應該還好，畢竟該image不再使用了）；

第二種是在判斷了該image未使用而且已足夠舊后，但是在os.remove刪除執行前，該image被使用到了，這就比較危險，該鎖應該就是前述

第一個鎖的作用。

log記錄

storage_register管理compute_nodes時的鎖：

2月 15 18:45:30 nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "storage-registry-lock" acquired by "nova.virt.storage_users.do_register_storage_use" :: waited 0.000s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327}}
2月 15 18:45:30 nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "storage-registry-lock" released by "nova.virt.storage_users.do_register_storage_use" :: held 0.001s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339}}
2月 15 18:45:30 nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "storage-registry-lock" acquired by "nova.virt.storage_users.do_get_storage_users" :: waited 0.000s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327}}
2月 15 18:45:30 nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "storage-registry-lock" released by "nova.virt.storage_users.do_get_storage_users" :: held 0.001s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339}}

計算、匹配active的base文件：

2月 15 18:04:23 nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Image id e0043447-2aa1-4c39-ae79-1e239ee519b3 yields fingerprint 1f629360db7c8ff0cde2a464f3784959aaced90c {{(pid=16212) _age_and_verify_cached_images}}
2月 15 18:04:23 nova-compute[16212]: INFO nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] image e0043447-2aa1-4c39-ae79-1e239ee519b3 at (nova/instances/_base/1f629360db7c8ff0cde2a464f3784959aaced90c): checking
2月 15 18:04:23 nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] image e0043447-2aa1-4c39-ae79-1e239ee519b3 at (/opt/stack/data/nova/instances/_base/1f629360db7c8ff0cde2a464f3784959aaced90c): image is in use {{(pid=16212) _mark_in_use}}
2月 15 18:04:23 nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Instance 9dbd9c9f-7d5b-4d1d-8ee3-4fdff68d2d70 is backed by 1f629360db7c8ff0cde2a464f3784959aaced90c {{(pid=16212) _list_backing_images}}
2月 15 18:04:23 nova-compute[16212]: INFO nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Active base files: /opt/stack/data/nova/instances/_base/1f629360db7c8ff0cde2a464f3784959aaced90c

刪除base鏡像的鎖操作：

2月 15 19:25:54 nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "aaa29360db7c8ff0cde2a464f3784959aaced90c" acquired by "nova.virt.libvirt.imagecache._inner_remove_old_enough_file" :: waited 0.000s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327}}
2月 15 19:25:54 nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "aaa29360db7c8ff0cde2a464f3784959aaced90c" released by "nova.virt.libvirt.imagecache._inner_remove_old_enough_file" :: held 0.001s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339}}

刪除swap文件的鎖操作：

2月 15 19:25:54 nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "swap_3" acquired by "nova.virt.libvirt.imagecache._inner_remove_old_enough_file" :: waited 0.000s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327}}
2月 15 19:25:54 nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "swap_3" released by "nova.virt.libvirt.imagecache._inner_remove_old_enough_file" :: held 0.001s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339}}

總體過程，在swap_3被刪掉后，鎖nova-swap_3留了下來：

2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_service.periodic_task [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Running periodic task ComputeManager._run_image_cache_manager_pass {{(pid=16212) run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:217}}
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "storage-registry-lock" acquired by "nova.virt.storage_users.do_register_storage_use" :: waited 0.001s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327}}
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "storage-registry-lock" released by "nova.virt.storage_users.do_register_storage_use" :: held 0.002s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339}}
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "storage-registry-lock" acquired by "nova.virt.storage_users.do_get_storage_users" :: waited 0.000s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327}}
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "storage-registry-lock" released by "nova.virt.storage_users.do_get_storage_users" :: held 0.001s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Adding swap_3 into backend swap images {{(pid=16212) _store_swap_image /opt/stack/nova/nova/virt/libvirt/imagecache.py:119}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Verify base images {{(pid=16212) _age_and_verify_cached_images /opt/stack/nova/nova/virt/libvirt/imagecache.py:348}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Image id  yields fingerprint da39a3ee5e6b4b0d3255bfef95601890afd80709 {{(pid=16212) _age_and_verify_cached_images /opt/stack/nova/nova/virt/libvirt/imagecache.py:355}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Image id e0043447-2aa1-4c39-ae79-1e239ee519b3 yields fingerprint 1f629360db7c8ff0cde2a464f3784959aaced90c {{(pid=16212) _age_and_verify_cached_images /opt/stack/nova/nova/virt/libvirt/imagecache.py:355}}
2月 15 19:25:54  nova-compute[16212]: INFO nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] image e0043447-2aa1-4c39-ae79-1e239ee519b3 at (/nova/instances/_base/1f629360db7c8ff0cde2a464f3784959aaced90c): checking
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] image e0043447-2aa1-4c39-ae79-1e239ee519b3 at (/nova/instances/_base/1f629360db7c8ff0cde2a464f3784959aaced90c): image is in use {{(pid=16212) _mark_in_use /opt/stack/nova/nova/virt/libvirt/imagecache.py:329}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] a5a58b25-36b3-4bbf-b33b-645559357347 is a valid instance name {{(pid=16212) _list_backing_images /opt/stack/nova/nova/virt/libvirt/imagecache.py:169}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] 9dbd9c9f-7d5b-4d1d-8ee3-4fdff68d2d70 is a valid instance name {{(pid=16212) _list_backing_images /opt/stack/nova/nova/virt/libvirt/imagecache.py:169}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] 9dbd9c9f-7d5b-4d1d-8ee3-4fdff68d2d70 has a disk file {{(pid=16212) _list_backing_images /opt/stack/nova/nova/virt/libvirt/imagecache.py:172}}
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.processutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Running cmd (subprocess): /usr/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /nova/instances/9dbd9c9f-7d5b-4d1d-8ee3-4fdff68d2d70/disk --force-share {{(pid=16212) execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372}}
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.processutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] CMD "/usr/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /nova/instances/9dbd9c9f-7d5b-4d1d-8ee3-4fdff68d2d70/disk --force-share" returned: 0 in 0.089s {{(pid=16212) execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Instance 9dbd9c9f-7d5b-4d1d-8ee3-4fdff68d2d70 is backed by 1f629360db7c8ff0cde2a464f3784959aaced90c {{(pid=16212) _list_backing_images /opt/stack/nova/nova/virt/libvirt/imagecache.py:187}}
2月 15 19:25:54  nova-compute[16212]: WARNING nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Unknown base file: /nova/instances/_base/aaa29360db7c8ff0cde2a464f3784959aaced90c
2月 15 19:25:54  nova-compute[16212]: INFO nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Active base files: /nova/instances/_base/1f629360db7c8ff0cde2a464f3784959aaced90c
2月 15 19:25:54  nova-compute[16212]: INFO nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Removable base files: /nova/instances/_base/aaa29360db7c8ff0cde2a464f3784959aaced90c
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "aaa29360db7c8ff0cde2a464f3784959aaced90c" acquired by "nova.virt.libvirt.imagecache._inner_remove_old_enough_file" :: waited 0.000s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327}}
2月 15 19:25:54  nova-compute[16212]: INFO nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Removing base or swap file: /nova/instances/_base/aaa29360db7c8ff0cde2a464f3784959aaced90c
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "aaa29360db7c8ff0cde2a464f3784959aaced90c" released by "nova.virt.libvirt.imagecache._inner_remove_old_enough_file" :: held 0.001s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Verification complete {{(pid=16212) _age_and_verify_cached_images /opt/stack/nova/nova/virt/libvirt/imagecache.py:384}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Verify swap images {{(pid=16212) _age_and_verify_swap_images /opt/stack/nova/nova/virt/libvirt/imagecache.py:333}}
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "swap_3" acquired by "nova.virt.libvirt.imagecache._inner_remove_old_enough_file" :: waited 0.000s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327}}
2月 15 19:25:54  nova-compute[16212]: INFO nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Removing base or swap file: /nova/instances/_base/swap_3
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "swap_3" released by "nova.virt.libvirt.imagecache._inner_remove_old_enough_file" :: held 0.001s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339}}