cinder側掛載卷流程分析

cinder側掛載卷流程分析，存儲類型以lvm+iscsi的方式為分析基礎
cinder側主要調用了三個接口
1)reserve_volume: 把volume的狀態改為attaching，阻止其它節點執行掛載操作。
2)initialize_connection: 這個方法負責構建和返回nova調用者需要的所有信息。返回的信息中包括CHAP credential, target-iqn 和lun 信息。
3)attach_volume: 把volume狀態改為in-use，掛載成功,並創建對應的attach記錄。

1、nova側調用cinder的reserve_volume方法

nova/volume/cinder.py
    @translate_volume_exception
    def reserve_volume(self, context, volume_id):
        cinderclient(context).volumes.reserve(volume_id)

1)cinderclient端接受到nova發送的reserve操作的http請求，其入口處理函數為

cinder/api/contrib/volume_actions.py:VolumeActionsController
    @wsgi.action('os-reserve')
    def _reserve(self, req, id, body):
        """Mark volume as reserved."""
        context = req.environ['cinder.context']
        # Not found exception will be handled at the wsgi level
        volume = self.volume_api.get(context, id)

        self.volume_api.reserve_volume(context, volume)
        return webob.Response(status_int=http_client.ACCEPTED)

該函數的主要作用是通過volume 的uuid，獲取volume實例信息，並調用volume目錄下的api模塊

2)進一步調用cinder volume的api模塊的reserve_volume函數，進行數據庫的操作，更新卷的狀態為attaching
該函數的主要作用是檢查指定的卷是否為available,如果卷的狀態是available，更新cinder數據庫，把卷的狀態標記為attaching來預留這個卷，防止其他api在別的地方使用這個卷
對於支持多路掛載的卷，有效狀態包括in-use

def reserve_volume(self, context, volume):
    expected = {'multiattach': volume.multiattach,
                'status': (('available', 'in-use') if volume.multiattach
                           else 'available')}

    result = volume.conditional_update({'status': 'attaching'}, expected)

    if not result:
        expected_status = utils.build_or_str(expected['status'])
        msg = _('Volume status must be %(expected)s to reserve, but the '
                'status is %(current)s.') % {'expected': expected_status,
                                             'current': volume.status}
        LOG.error(msg)
        raise exception.InvalidVolume(reason=msg)

    LOG.info(_LI("Reserve volume completed successfully."),
             resource=volume)

2、nova側向cinder發送initialize_connection請求，請求獲取卷的所有連接信息

nova/virt/block_device.py:DriverVolumeBlockDevice
    def attach(self, context, instance, volume_api, virt_driver,
               do_check_attach=True, do_driver_attach=False, **kwargs):
        volume = volume_api.get(context, self.volume_id)
        if do_check_attach:
            volume_api.check_attach(context, volume, instance=instance)

        volume_id = volume['id']
        context = context.elevated()

        connector = virt_driver.get_volume_connector(instance)
        connection_info = volume_api.initialize_connection(context,
                                                           volume_id,
                                                           connector)
        if 'serial' not in connection_info:
            connection_info['serial'] = self.volume_id
        self._preserve_multipath_id(connection_info)
        ........
    

1)cinderclient接受nova發送過來的os-initialize_connection請求

@wsgi.action('os-initialize_connection')
def _initialize_connection(self, req, id, body):
    """Initialize volume attachment."""
    context = req.environ['cinder.context']
    # Not found exception will be handled at the wsgi level
    volume = self.volume_api.get(context, id)
    try:
        connector = body['os-initialize_connection']['connector']
    except KeyError:
        raise webob.exc.HTTPBadRequest(
            explanation=_("Must specify 'connector'"))
    try:
        info = self.volume_api.initialize_connection(context,volume,connector)    
        ....

2)進一步調用volume目錄下的api模塊的initialize_connection函數,對該請求進行處理

cinder/volume/api.py:API類
    @wrap_check_policy
    def initialize_connection(self, context, volume, connector):
        if volume.status == 'maintenance':
            LOG.info(_LI('Unable to initialize the connection for '
                         'volume, because it is in '
                         'maintenance.'), resource=volume)
            msg = _("The volume connection cannot be initialized in "
                    "maintenance mode.")
            raise exception.InvalidVolume(reason=msg)
        init_results = self.volume_rpcapi.initialize_connection(context,
                                                                volume,
                                                                connector)
        LOG.info(_LI("Initialize volume connection completed successfully."),
                 resource=volume)
        return init_results

3)cinder api進一步發送RPC請求給volume所在的cinder-volume服務節點，最終在cinder-volume節點，
由cinder/volume/manager.py:VolumeManager的initialize_connection處理，該函數的處理，主要包括如下內容

   def initialize_connection(self, context, volume, connector):
         ....
           utils.require_driver_initialized(self.driver)
            step 1: self.driver.validate_connector(connector)
            step 2: model_update = self.driver.create_export(context.elevated(),volume, connector)
            step 3: volume.update(model_update)
            setp 4: conn_info = self.driver.initialize_connection(volume, connector)
        return conn_info

step 1:
對於LVM + iSCSI方式，validate_connector就是檢查有沒有initiator字段，即nova-compute節點的initiator信息
代碼跳轉過程如下：drivers/lvm.py -> targets/lio.py -> targets/iscsi.py。

cinder/volume/targets/iscsi.py:ISCSITarget
def validate_connector(self, connector):
    # NOTE(jdg): api passes in connector which is initiator info
    if 'initiator' not in connector:
        err_msg = (_LE('The volume driver requires the iSCSI initiator '
                       'name in the connector.'))
        LOG.error(err_msg)
        raise exception.InvalidConnectorException(missing='initiator')
    return True

step 2 :調用cinder-rtstool工具創建target，並把卷volume添加到target中創建出lun,認證信息。

def create_export(self, context, volume, connector, vg=None):
    if vg is None:
        vg = self.configuration.volume_group
    volume_path = "/dev/%s/%s" % (vg, volume['name'])
    export_info = self.target_driver.create_export(
        context,
        volume,
        volume_path)
    return {'provider_location': export_info['location'],
            'provider_auth': export_info['auth'], }

最終調用的是cinder/volume/targets/iscsi.py:ISCSITarget類

    def create_export(self, context, volume, volume_path):
        """Creates an export for a logical volume."""
        # 'iscsi_name': 'iqn.2010-10.org.openstack:volume-00000001'
        iscsi_name = "%s%s" % (self.configuration.iscsi_target_prefix,------設置iscsi name,形式為iqn.2010-10.org.openstack:volume-uuid
                               volume['name'])
        iscsi_target, lun = self._get_target_and_lun(context, volume)---返回target，和lun的編號，值為(0,0)

        # Verify we haven't setup a CHAP creds file already
        # if DNE no big deal, we'll just create it
        chap_auth = self._get_target_chap_auth(context, volume)------從數據庫volumes表中，讀取該卷的provider_auth字段，獲取認證信息，若沒有，則創建
        if not chap_auth:
            chap_auth = (vutils.generate_username(),-----創建auth認證信息
                         vutils.generate_password())

        # Get portals ips and port
        portals_config = self._get_portals_config()-------獲取portals配置，該函數返回的字典格式如下 {'portals_ips': portals_ips,'portals_port': self.configuration.iscsi_port}

        # NOTE(jdg): For TgtAdm case iscsi_name is the ONLY param we need
        # should clean this all up at some point in the future
        tid = self.create_iscsi_target(iscsi_name,-----------------創建target
                                       iscsi_target,
                                       lun,
                                       volume_path,
                                       chap_auth,
                                       **portals_config)
        data = {}
        data['location'] = self._iscsi_location(
            self.configuration.iscsi_ip_address, tid, iscsi_name, lun,
            self.configuration.iscsi_secondary_ip_addresses)
        LOG.debug('Set provider_location to: %s', data['location'])
        data['auth'] = self._iscsi_authentication(
            'CHAP', *chap_auth)
        return data

創建target操作，調用的是cinder/volume/targets/lio.py中的create_iscsi_target方法
這個函數下發的參數為
name：iqn.2010-10.org.openstack:volume-uuid
tid:0
lun:0
path:卷的路徑
chap_auth:{username,passowrd}
kwargs:{'portals_ips': portals_ips,存儲服務器的ip
'portals_port': self.configuration.iscsi_port,一般是3260
}

def create_iscsi_target(self, name, tid, lun, path,chap_auth=None, **kwargs):
    # tid and lun are not used
    vol_id = name.split(':')[1]
    LOG.info(_LI('Creating iscsi_target for volume: %s'), vol_id)
    chap_auth_userid = ""
    chap_auth_password = ""
    if chap_auth is not None:
        (chap_auth_userid, chap_auth_password) = chap_auth

    optional_args = []
    if 'portals_port' in kwargs:
        optional_args.append('-p%s' % kwargs['portals_port'])

    if 'portals_ips' in kwargs:
        optional_args.append('-a' + ','.join(kwargs['portals_ips']))

    try:
        command_args = ['cinder-rtstool',
                        'create',
                        path,
                        name,
                        chap_auth_userid,
                        chap_auth_password,
                        self.iscsi_protocol == 'iser'] + optional_args
        self._execute(*command_args, run_as_root=True)
    except putils.ProcessExecutionError:
        LOG.exception(_LE("Failed to create iscsi target for volume "
                          "id:%s."), vol_id)

        raise exception.ISCSITargetCreateFailed(volume_id=vol_id)

    iqn = '%s%s' % (self.iscsi_target_prefix, vol_id)
    tid = self._get_target(iqn)
    if tid is None:
        LOG.error(_LE("Failed to create iscsi target for volume "
                      "id:%s."), vol_id)
        raise exception.NotFound()

    # We make changes persistent
    self._persist_configuration(vol_id)
    return tid

def _iscsi_location(self, ip, target, iqn, lun=None, ip_secondary=None):
    ip_secondary = ip_secondary or []
    port = self.configuration.iscsi_port
    portals = map(lambda x: "%s:%s" % (x, port), [ip] + ip_secondary)
    return ("%(portals)s,%(target)s %(iqn)s %(lun)s"
            % ({'portals': ";".join(portals),
                'target': target, 'iqn': iqn, 'lun': lun}))

step 3:創建完target以后，更新cinder數據庫volumes表中，該volume的provider_location，provider_auth兩個字段的值

step 4：調用cinder-rtstool的add-initiator子命令，把計算節點的initiator增加到剛剛創建的target acls中,並把所有的信息拼裝返回給nova使用。

cinder/volume/targets/lio.py：
   def initialize_connection(self, volume, connector):
        volume_iqn = volume['provider_location'].split(' ')[1]
        (auth_method, auth_user, auth_pass) = \
            volume['provider_auth'].split(' ', 3)
        # Add initiator iqns to target ACL
        try:
            self._execute('cinder-rtstool', 'add-initiator',
                          volume_iqn,
                          auth_user,
                          auth_pass,
                          connector['initiator'],
                          run_as_root=True)
        ......
        return super(LioAdm, self).initialize_connection(volume, connector)

3、nova給cinderclient發送attach_volume命令，更改cinder數據庫中，volume狀態

nova/virt/block_device.py:API
    @translate_volume_exception
    def attach(self, context, volume_id, instance_uuid, mountpoint, mode='rw'):
        cinderclient(context).volumes.attach(volume_id, instance_uuid,
                                             mountpoint, mode=mode)

1)cinder側接受nova更新cinder數據庫的入口函數

cinder/api/contrib/volume_actions.py
    @wsgi.action('os-attach')
    def _attach(self, req, id, body):
            .....
            self.volume_api.attach(context, volume,instance_uuid, host_name, mountpoint, mode)
            ....

2)最后cinder-api通過RPC請求到cinder-volume節點，更新數據庫，把volume狀態改為in-use，並創建對應的attach記錄。

cinder/volume/manager.py:VolumeManager
    def attach_volume(self, context, volume_id, instance_uuid, host_name,
                      mountpoint, mode, volume=None):
        """Updates db to show volume is attached.""
   ......
     attachment = volume.begin_attach(mode
    ......