cinder create volume的流程(1)

前提：代碼的跟蹤，使用的是ocata版本

零、執行cinder create 命令，創建數據卷，打開debug開關

[root@osnode241001 ~]# cinder --debug create --display-name=chenwei 1
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://11.27.241.251:5000/v2.0/ -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
DEBUG:keystoneclient.session:RESP: [200] CaseInsensitiveDict({'date': 'Tue, 22 Jan 2019 06:25:22 GMT', 
'vary': 'X-Auth-Token', 'content-length': '423', 'content-type': 'application/json', 'connection': 'close'}) 
RESP BODY: {"version": 
{"status": "stable", 
"updated": "2014-04-17T00:00:00Z", 
"media-types": [{"base": "application/json", 
"type": "application/vnd.openstack.identity-v2.0+json"}, 
{"base": "application/xml", 
"type": "application/vnd.openstack.identity-v2.0+xml"}], 
"id": "v2.0", 
"links": [{"href": "http://10.27.241.251:5000/v2.0/", "rel": "self"}, 
{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://11.27.241.251:5000/v2.0/tokens
DEBUG:keystoneclient.session:REQ: curl -i -X POST http://11.27.241.251:8776/v1/3333f85b0b784f7cbbb21755e932229b/volumes 
-H "User-Agent: python-cinderclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: TOKEN_REDACTED" 
-d '{"volume": {
"status": "creating", 
"availability_zone": null, 
"source_volid": null, 
"display_description": null, 
"snapshot_id": null, 
"user_id": null, 
"size": 1, 
"display_name": "chenwei", 
"imageRef": null, 
"attach_status": "detached", 
"volume_type": null, 
"project_id": null, 
"metadata": {}}
}'
DEBUG:keystoneclient.session:RESP: [200] CaseInsensitiveDict({'content-length': '362', 
'x-compute-request-id': 'req-931befc0-e7e7-4e0e-bd57-225e3320eb35', 
'connection': 'close', 'date': 'Tue, 22 Jan 2019 06:25:23 GMT', 
'content-type': 'application/json', 'x-openstack-request-id': 'req-931befc0-e7e7-4e0e-bd57-225e3320eb35'}) 
RESP BODY: {"volume": {"status": "creating", 
"display_name": "chenwei", "attachments": [], 
"availability_zone": "nova", "bootable": "false",
 "encrypted": false, "created_at": "2019-01-22T06:25:23.454840", 
"display_description": null, "volume_type": "None", 
"snapshot_id": null, "source_volid": null, "metadata": {}, "id": "dd9dab6b-3b09-4b28-b782-77fdeefd099d", "size": 1}}

+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2019-01-22T06:25:23.454840      |
| display_description |                 None                 |
|     display_name    |               chenwei                |
|      encrypted      |                False                 |
|          id         | dd9dab6b-3b09-4b28-b782-77fdeefd099d |
|       metadata      |                  {}                  |
|         size        |                  1                   |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+

一、cinder-api流程部分　

cinder-api接收到以http協議發送的POST請求，經過http body分析：創建一個size為1GB的volume
進入文件cinder/api/v1/volumes.py調用class VolumeController(wsgi.Controller):類的create方法

def create(self, req, body)
    前半部分是對cinder request請求體里面的內容進行獲取和驗證
	#最主要代碼
	new_volume = self.volume_api.create(context,
            size,
            volume.get('display_name'),
            volume.get('display_description'),
            **kwargs)

進入cinder/volume/api.py文件的class API(base.Base)類create方法　　

class API(base.Base):
    def create(self, context, size, name, description, snapshot=None,
               image_id=None, volume_type=None, metadata=None,
               availability_zone=None, source_volume=None,
               scheduler_hints=None,
               source_replica=None, consistencygroup=None,
               cgsnapshot=None, multiattach=False, source_cg=None,
               group=None, group_snapshot=None, source_group=None):
    #檢查從鏡像創建卷還是新建
    check_policy(context, 'create_from_image' if image_id else 'create')
    #當從鏡像或者卷創建新卷時，新卷的大小如果未明確指定，那么默認取原來卷的值
    #cinderclient下發的volume的大小是字符串形式，而有些cloud調用是直接調用api，所以需要對size的類型進行檢查
    前半部分主要邏輯是對要創建的卷所需的參數進一步驗證，例如：卷可用域，卷的類型檢查等
    #最主要邏輯代碼
            sched_rpcapi = (self.scheduler_rpcapi if (
                            not cgsnapshot and not source_cg and
                            not group_snapshot and not source_group)
                            else None)
            volume_rpcapi = (self.volume_rpcapi if (
                             not cgsnapshot and not source_cg and
                             not group_snapshot and not source_group)
                             else None)
    #進入創建卷的工作流
    #cinder-api啟動volume_create_api工作流（flow），flow的狀態從pending->running->success，
    #該工作流包含若干個任務（TASK），TASK的轉換狀態與工作流一樣，都是到success結束，每個TASK完成特定的任務
            flow_engine = create_volume.get_flow(self.db,
                                                 self.image_service,
                                                 availability_zones,
                                                 create_what,
                                                 sched_rpcapi,
                                                 volume_rpcapi)    

進入cinder/volume/flows/api/create_volume.py文件中，調用get_flow方法

def get_flow(db_api, image_service_api, availability_zones, create_what,
             scheduler_rpcapi=None, volume_rpcapi=None):
    """Constructs and returns the api entrypoint flow.
     構造並返回api entrypoint 工作流
    This flow will do the following:

    1. Inject keys & values for dependent tasks.
    為依賴任務注入的鍵和值。
    2. Extracts and validates the input keys & values.
    提取並驗證輸入的鍵和值
    3. Reserves the quota (reverts quota on any failures).
    保留配額(在發生故障時恢復配額)
    4. Creates the database entry.
    創建數據庫條目
    5. Commits the quota.
    提交配額
    6. Casts to volume manager or scheduler for further processing.
    將數據類型轉換到volume manager or scheduler以進行進一步處理
    """    
    創建api線性流
    api_flow = linear_flow.Flow(flow_name)
    
    #依次添加ExtractVolumeRequestTask、QuotaReserveTask、EntryCreateTask、QuotaCommitTask
    api_flow.add(ExtractVolumeRequestTask(
        image_service_api,
        availability_zones,
        rebind={'size': 'raw_size',
                'availability_zone': 'raw_availability_zone',
                'volume_type': 'raw_volume_type'}))
    api_flow.add(QuotaReserveTask(),
                 EntryCreateTask(),
                 QuotaCommitTask())    
    if scheduler_rpcapi and volume_rpcapi:
        # This will cast it out to either the scheduler or volume manager via
        # the rpc apis provided.
        api_flow.add(VolumeCastTask(scheduler_rpcapi, volume_rpcapi, db_api))

    # Now load (but do not run) the flow using the provided initial data.
    return taskflow.engines.load(api_flow, store=create_what)

volume_create_api工作流包含的TASK：
1)ExtractVolumeRequestTask獲取request信息
2)QuotaReserverTask預留配額
3)EntryCreateTask在數據庫中創建volume條目
4)QuotaCommitTask確認配額
1-4都是創建volume的准備工作
5)VolumeCastTask向cinder-scheduler發送消息，實際就是把消息發給rabbitmq，
最后，volume_create_api flow達到success狀態，完成了pending->running->success整個流程。
這里的Volume created successfully只代表cinder-api處理完成，不代表后端存儲節點上的volume創建成功

VolumeCastTask類的功能
class VolumeCastTask(flow_utils.CinderTask):
    """Performs a volume create cast to the scheduler or to the volume manager.
    向 scheduler or to the volume manager發送一個創建卷的cast
    This will signal a transition of the api workflow to another child and/or
    related workflow on another component.
    將api工作流轉換為另一個子工作流和/或另一個組件上的相關工作流
    Reversion strategy: rollback source volume status and error out newly
    created volume.
    """
    def __init__(self, scheduler_rpcapi, volume_rpcapi, db):
        requires = ['image_id', 'scheduler_hints', 'snapshot_id',
                    'source_volid', 'volume_id', 'volume', 'volume_type',
                    'volume_properties', 'source_replicaid',
                    'consistencygroup_id', 'cgsnapshot_id', 'group_id', ]
        super(VolumeCastTask, self).__init__(addons=[ACTION],
                                             requires=requires)
        self.volume_rpcapi = volume_rpcapi #會進入cinder/volume/rpcapi.py文件中create_volume方法
        self.scheduler_rpcapi = scheduler_rpcapi #會進入cinder/scheduler/rpcapi.py 文件中create_volume方法
        self.db = db

二、cinder-scheduler流程部分

進入cinder/scheduler/rpcapi.py文件中class SchedulerAPI類的create_volume方法

class SchedulerAPI(rpc.RPCAPI)類的
    def create_volume(self, ctxt, volume, snapshot_id=None, image_id=None,
                      request_spec=None, filter_properties=None):
        volume.create_worker()
        cctxt = self._get_cctxt()
        msg_args = {'snapshot_id': snapshot_id, 'image_id': image_id,
                    'request_spec': request_spec,
                    'filter_properties': filter_properties, 'volume': volume}
        return cctxt.cast(ctxt, 'create_volume', **msg_args)

自動映射到cinder/scheduler/manager.py中的 create_volume 方法

class SchedulerManager類
    def create_volume(self, context, volume, snapshot_id=None, image_id=None,
                      request_spec=None, filter_properties=None):
        self._wait_for_scheduler()

        try:
        #cinder-scheduler開啟一個volume_create_scheduler flow工作流
            flow_engine = create_volume.get_flow(context,
                                                 self.driver,
                                                 request_spec,
                                                 filter_properties,
                                                 volume,
                                                 snapshot_id,
                                                 image_id)
        except Exception:
            msg = _("Failed to create scheduler manager volume flow")
            LOG.exception(msg)
            raise exception.CinderException(msg)

        with flow_utils.DynamicLogListener(flow_engine, logger=LOG):

進入到cinder/scheduler/flows/create_volume.py 文件中調用get_flow函數

def get_flow(context, driver_api, request_spec=None,
             filter_properties=None,
             volume=None, snapshot_id=None, image_id=None):

    """Constructs and returns the scheduler entrypoint flow.
        構造並返回scheduler entrypoint 工作流
    This flow will do the following:
    這個流主要做如下內容：
    1. Inject keys & values for dependent tasks.
    為依賴任務注入鍵和值
    2. Extract a scheduler specification from the provided inputs.
    從提供的輸入中提取scheduler規格參數
    3. Use provided scheduler driver to select host and pass volume creation request further.
    使用提供的調度程序驅動程序選擇主機並傳遞卷創建請求
    創建volume_create_scheduler 線性工作流
    flow_name = ACTION.replace(":", "_") + "_scheduler"
    scheduler_flow = linear_flow.Flow(flow_name)    
    
    # This will extract and clean the spec from the starting values.
    這將從初始值中提取和清除規范
    scheduler_flow.add(ExtractSchedulerSpecTask(
        rebind={'request_spec': 'raw_request_spec'}))

    # This will activate the desired scheduler driver (and handle any
    # driver related failures appropriately).
    這將激活期望的調度驅動程序
    scheduler_flow.add(ScheduleCreateVolumeTask(driver_api))

    # Now load (but do not run) the flow using the provided initial data.
    return taskflow.engines.load(scheduler_flow, store=create_what)

volume_create_scheduler flow 工作流,包含兩個任務
ExtractSchedulerSpecTask
ScheduleCreateVolumeTask 完成filter和weight的工作
filter依次經過：
AvailabilityZoneFilter，可以給存儲節點划分az，這樣就可以通過該過濾器過濾到用戶指定的az中創建卷；
CapacityFilter過濾掉存儲空間不滿足用戶需求的存儲節點；
CapabilitiesFilter，不同的后端存儲，比如lvm、ceph、sheepdog等等這些產品有不同的特性，
cinder用戶允許創建volume時指定需要的Capabilities（特性）,通過volume Type來指定這個特性，volume Type可以根據不同的存儲產品特性自定義

最后volume_create_scheduler工作流到達success狀態

class ScheduleCreateVolumeTask(flow_utils.CinderTask):
    def __init__(self, driver_api, **kwargs):
        super(ScheduleCreateVolumeTask, self).__init__(addons=[ACTION],
                                                       **kwargs)
        self.driver_api = driver_api
        self.message_api = message_api.API()
        
進入到cinder/scheduler/driver.py
class Scheduler(object):
所有調度類都需要繼承的基類
    """The base class that all Scheduler classes should inherit from."""

    def __init__(self):
        self.host_manager = importutils.import_object(
            CONF.scheduler_host_manager)
        self.volume_rpcapi = volume_rpcapi.VolumeAPI() #會調用cinder/volume/rpcapi.py的class VolumeAPI(rpc.RPCAPI)
        

cinder-scheduler發送消息給cinder-volume,讓其創建volume.

進入 cinder/scheduler/filter_scheduler.py,方法為class FilterScheduler(driver.Scheduler):

 def schedule_create_volume(self, context, request_spec, filter_properties):
  #主要實現是
        self.volume_rpcapi.create_volume(context, updated_volume, request_spec,
                                         filter_properties,
                                         allow_reschedule=True) 

三、cinder-volume流程部分

進入cinder/volume/rpcapi.py文件中調用create_volume方法 rpc調用

    def create_volume(self, ctxt, volume, request_spec, filter_properties,
                      allow_reschedule=True):
        cctxt = self._get_cctxt(volume.service_topic_queue)
        cctxt.cast(ctxt, 'create_volume',
                   request_spec=request_spec,
                   filter_properties=filter_properties,
                   allow_reschedule=allow_reschedule,
                   volume=volume)

自動映射到cinder/volume/manager.py 文件中類VolumeManager中的create_volume方法

   @objects.Volume.set_workers
    def create_volume(self, context, volume, request_spec=None,
                      filter_properties=None, allow_reschedule=True):
        """Creates the volume."""
        #最主要的是創建卷工作流程
        cinder-volume開啟volume_create_manager flow工作流
            flow_engine = create_volume.get_flow(
                context_elevated,
                self,
                self.db,
                self.driver,
                self.scheduler_rpcapi,
                self.host,
                volume,
                allow_reschedule,
                context,
                request_spec,
                filter_properties,
                image_volume_cache=self.image_volume_cache,
            )    

進入cinder/volume/flows\manager/create_volume.py文件中調用get_flow方法

def get_flow(context, manager, db, driver, scheduler_rpcapi, host, volume,
             allow_reschedule, reschedule_context, request_spec,
             filter_properties, image_volume_cache=None):

    """Constructs and returns the manager entrypoint flow.
    構造並返回manager entrypoint工作流
    This flow will do the following:
    1. Determines if rescheduling is enabled (ahead of time).
    確定是否啟用了重新調度(提前)
    2. Inject keys & values for dependent tasks.
    為依賴任務注入鍵和值
    3. Selects 1 of 2 activated only on *failure* tasks (one to update the db
       status & notify or one to update the db status & notify & *reschedule*).
       在僅“失敗”任務上，選擇激活的2個選項中的1個，一個是更新數據庫狀態和通知
       另一個是更新數據庫狀態，通知和重新調度
    4. Extracts a volume specification from the provided inputs.
    從提供的輸入中提取卷規格參數
    5. Notifies that the volume has started to be created.
    通知卷已開始創建
    6. Creates a volume from the extracted volume specification.
    從提取的卷規范中創建卷
    7. Attaches a on-success *only* task that notifies that the volume creation
       has ended and performs further database status updates.    
    附加一個on-success“only”任務，該任務通知卷的創建已結束並執行進一步的數據庫狀態更新    
    
    #volume_create_manager flow線性工作流
    flow_name = ACTION.replace(":", "_") + "_manager"
    volume_flow = linear_flow.Flow(flow_name)    
    volume_flow.add(ExtractVolumeRefTask(db, host, set_error=False))

    retry = filter_properties.get('retry', None)

    # Always add OnFailureRescheduleTask and we handle the change of volume's
    # status when reverting the flow. Meanwhile, no need to revert process of
    # ExtractVolumeRefTask.
    do_reschedule = allow_reschedule and request_spec and retry
    volume_flow.add(OnFailureRescheduleTask(reschedule_context, db, driver,
                                            scheduler_rpcapi, do_reschedule))

    LOG.debug("Volume reschedule parameters: %(allow)s "
              "retry: %(retry)s", {'allow': allow_reschedule, 'retry': retry})

    volume_flow.add(ExtractVolumeSpecTask(db),
                    NotifyVolumeActionTask(db, "create.start"),
                    CreateVolumeFromSpecTask(manager,
                                             db,
                                             driver,
                                             image_volume_cache),
                    CreateVolumeOnFinishTask(db, "create.end"))

    # Now load (but do not run) the flow using the provided initial data.
    return taskflow.engines.load(volume_flow, store=create_what)

該volume_create_manager flow的Task包括：
1)ExtractVolumeRefTask
2)OnFailureRescheduleTask
3)ExtractVolumeSpecTask
4)NotifyVolumeActionTask
這四個Task為volume創建做准備
5)CreateVolumeFromSpecTask
真正的執行volume創建任務，openstack默認的是lvm，可以在cinder.conf里指定后端存儲類型，當然就用lvcreate這樣的命令創建卷了，
實際使用sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate 命令行，

6)CreateVolumeOnFinishTask
CreatevolumeOnFinishTask，完成掃尾工作
最后volume_create_manager flow達到success狀態

參考文章：

https://blog.csdn.net/u010855924/article/details/60881917