碼上快樂

相關內容    簡體    繁體

Openstack平台虛擬機疏散失敗提示(pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')問題

本文轉載自   查看原文   2020-05-15 10:16   147    cinder/ mysql連接失敗/ OpenStack/ openstack

一.問題的表象

過程:創建完windowserver2012r2版本虛擬機，掛載cinder卷，寫入一定的數據
問題:關閉虛擬機所在的宿主機，執行疏散操作，虛擬機疏散失敗

二.問題日志

  • 遷移的目的服務器nova-compute的日志信息如下：
2018-11-28 10:12:01.120 7 ERROR nova.volume.cinder [req-8d60395f-5c21-4699-bd5e-31b921acb378 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - default default] Delete attachment failed for attachment e6b83198-8286-4925-b1d8-1b09b0ae9f79. Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6) Code: 500: ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server [req-8d60395f-5c21-4699-bd5e-31b921acb378 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - default default] Exception during message handling: ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 7829, in detach_volume
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/utils.py", line 976, in decorated_function
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in decorated_function
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in decorated_function
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 5436, in detach_volume
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id=attachment_id)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 5389, in _detach_volume
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id=attachment_id, destroy_bdm=destroy_bdm)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 415, in detach
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id, destroy_bdm)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 412, in _do_locked_detach
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     self._do_detach(*args, **_kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 400, in _do_detach
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     volume_api.attachment_delete(context, self['attachment_id'])
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 379, in wrapper
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     res = method(self, ctx, *args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 415, in wrapper
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     res = method(self, ctx, attachment_id, *args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 838, in attachment_delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     'code': getattr(ex, 'code', None)})
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 831, in attachment_delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/v3/attachments.py", line 39, in delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return self._delete("/attachments/%s" % base.getid(attachment))
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/base.py", line 339, in _delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     resp, body = self.api.client.delete(url)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/client.py", line 209, in delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return self._cs_request(url, 'DELETE', **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/client.py", line 191, in _cs_request
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return self.request(url, method, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/client.py", line 177, in request
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     raise exceptions.from_response(resp, body)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server
  • 遷移的目的服務器cinder-volume的日志信息如下：
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server [req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - - -] Exception during message handling: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py", line 4503, in attachment_delete
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     attachment_id)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/objects/base.py", line 314, in get_by_id
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     orm_obj = db.get_by_id(context, cls.model, id, *args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/api.py", line 1777, in get_by_id
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return IMPL.get_by_id(context, model, id, *args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 182, in wrapper
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 6980, in get_by_id
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return _GET_METHODS[model](context, id, *args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 182, in wrapper
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 2050, in volume_attachment_get
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return _attachment_get(context, attachment_id)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 1982, in _attachment_get
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     .options(joinedload('volume'))
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     ret = list(self[0:1])
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return list(res)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return self._execute_and_instances(context)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     close_with_result=True)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     **kw
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     conn = self.session.connection(**kw)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     execution_options=execution_options)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     conn = engine.contextual_connect(**kw)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self._wrap_pool_connect(self.pool.connect, None),
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     e, dialect, self)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     util.raise_from_cause(newraise, exc_info)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return fn()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return _ConnectionFairy._checkout(self)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     fairy = _ConnectionRecord.checkout(pool)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 532, in checkout
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     rec = pool._do_get()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1186, in _do_get
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self._dec_overflow()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     compat.reraise(exc_type, exc_value, exc_tb)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1183, in _do_get
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return self._create_connection()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 350, in _create_connection
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return _ConnectionRecord(self)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 477, in __init__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self.__connect(first_connect_check=True)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     connection = pool._invoke_creator(self)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return dialect.connect(*cargs, **cparams)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return self.dbapi.connect(*cargs, **cparams)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return Connection(*args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 699, in __init__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self.connect()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 936, in connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self._request_authentication()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1156, in _request_authentication
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     auth_packet = self._read_packet()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     packet_header = self._read_bytes(4)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1037, in _read_bytes
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server

日志分析:
1.我們模擬的是虛擬機的疏散過程，在疏散的過程中，我們查看到目的服務器的nova-compute.log和cinder-volume.log的日志存在錯誤

2.進一步分析得知，我們發現出現問題的原因在cinder-volume，對比虛擬機能夠正常疏散時的log日志，我們發現正常疏散時，cinder-volume.log的疏散顯示的日志如下：

2018-11-28 14:37:54.772 34 INFO cinder.volume.manager [req-e287af5e-c5c3-4dc3-9cba-ab003ad12704 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - - -] Terminate volume connection completed successfully.
2018-11-28 14:37:54.773 34 DEBUG cinder.volume.manager [req-e287af5e-c5c3-4dc3-9cba-ab003ad12704 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - - -] Deleting attachment 6e32750e-6134-45cc-942f-c1065826f28b. _do_attachment_delete /var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py:4519

3.對比cinder-volume.log問題日志和正常日志，我們分析，在虛擬機的疏散過程，涉及卷的卸載過程，懷疑的cinder卷的卸載存在問題，對此，我模擬windows server2012的虛擬機掛載cinder卷，寫大量數據，然后分離卷連接，模擬了大概50次左右，復現過一次cinder卷分離失敗的錯誤，對比log進行分析對比，發現和上面的虛擬機疏散失敗問題一致。

4.分析cinder卷的掛載流程，我們發現根據log提示和實際cinder卷的卸載流程存在中斷卷連接的操作，也就是第三章節中1.4.6.2步驟。這里我們第三章節從cinder-volume的日志來看，函數在/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1037, in _read_bytes此處提示丟失數據庫連接錯誤，導致了中斷卷連接時報錯。

三.卷的卸載流程


上述流程圖顯示，卷在detach()->get vol info by id->terminate connection(call)時報錯。

代碼分析：

我們看到函數在_read_bytes處拋出錯誤：

進入cinder-volume容器，查看/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py代碼的此函數如下:

def _read_bytes(self, num_bytes):
        self._sock.settimeout(self._read_timeout)
        while True:
            try:
                data = self._rfile.read(num_bytes)
                break
            except (IOError, OSError) as e:
                if e.errno == errno.EINTR:
                    continue
                self._force_close()
                raise err.OperationalError(
                    CR.CR_SERVER_LOST,
                    "Lost connection to MySQL server during query (%s)" % (e,))
        #如果MySQL服務器沒有根據協議所述發送足夠的數據，就會拋出如下錯誤
        if len(data) < num_bytes:
            self._force_close()
            raise err.OperationalError(
                CR.CR_SERVER_LOST, "Lost connection to MySQL server during queryy
")
        return data
        
#當服務器正在關閉(但顯然仍在運行，因為這是從服務器發送的數據)時，將在專門為MariaDB記錄的情況下拋出     
if packet_number == 0:
# MariaDB sends error packet with seqno==0 when shutdown
    raise err.OperationalError(
    CR.CR_SERVER_LOST,
    "Lost connection to MySQL server during query")
raise err.InternalError(
    "Packet sequence number wrong - got %d expected %d"
    % (packet_number, self._next_seq_id))

四.解決辦法

MySQL數據庫中，max_allowed_packet允許的server段接受的數據包最大為4M，有的時候大的插入和更新可能會被max_allowed_packet參數限制，導致失敗，這里我們修改了max_allowed_packet參數。

[root@hci-test01 templates]# vi /usr/share/kolla-ansible/ansible/roles/mariadb/templates/galera.cnf.j2
[server]
#增加如下行
max_allowed_packet= '32M'

相關知識鏈接:

數據庫知識: https://blog.csdn.net/hxpjava1/article/details/80522201
數據庫問題分析：https://bugs.launchpad.net/tripleo/+bug/1691951
cinder的流程圖：https://blog.csdn.net/inodes/article/details/24729883


免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。



猜您在找 解決python pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query') pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query') pymysql報錯OperationalError: (2013, 'Lost connection to MySQL server during query') ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) _mysql_exceptions.OperationalError: (2013, 'Lost connection to MySQL server during query') flask_sqlalchemy 2013 Lost connection to MySQL server during query Error NO.2013 Lost connection to Mysql server during query (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'localhost' ([Errno 61] Connection refused)") 1、pymysql.err.OperationalError: (2003, “Can’t connect to MySQL server on ‘localhost’ ([Errno 111] Connection refused)”) [奇葩問題] ERROR 2013 (HY000): Lost connection to MySQL server during query
 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM