dubbo 報錯問題記錄:may be version or group mismatch

      凌晨時候釘釘告警群里一直大量報錯：

      

   接口異常報警:項目:mp-rest,域名:inside-mp.01zhuanche.com,IP:10.30.3.60,接口地址:/api/v3/driverLogin/driverType,請求方式:POST,錯誤信息:com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method findByPhone in the service com.zhuanche.driver.service.DriverInfoService. Tried 1 times of the providers [10.30.3.72:8080] (1/8) from the registry 10.0.7.56:2181 on the consumer 10.30.3.60 using the dubbo version 2.5.8. Last error is: Failed to invoke remote method: findByPhone, provider: dubbo://10.30.3.72:8080/com.zhuanche.driver.service.DriverInfoService?anyhost=true&application=mp-rest&check=false&default.accesslog=/u01/mp-driver-provider/log/access.log&default.check=false&default.loadbalance=leastactive&default.retries=0&dispatcher=message&dubbo=2.5.8&generic=false&interface=com.zhuanche.driver.service.DriverInfoService&logger=slf4j&methods=queryReatImeiByDriverIdAndTime,updateImeiAndAppversion,findByIdcardOrPlateNum,updatePhoneByDriverId,findHistoryDriver,listByLicensePlates,sendDriverToMq,findDriverBaseInfoByPhone,findDriverInfoBySupplierId,findByPhone,getValidateLoginByDriverId,findChatUserIdByDriverId,findPhoneByCityIdOrCooperation,findHistoryBylicensePlates,findDriverIdByChatUserId,resetImei,findYOTHistoryDriver,findDriverBylicensePlates,findByDriverId,findYOTHistoryBylicensePlates,searchBlackDriver,updateIMEIByDriverId,updateGroupIdByDriverId,countReatImeiByDriverIdAndTime&organization=zhuanche&owner=mp&pid=8&register.ip=10.30.3.60&remote.timestamp=1561644769858&revision=0.5.1-20190505.093234-1&side=consumer&timeout=5000&timestamp=1563247918842, cause: com.alibaba.dubbo.remoting.RemotingException: Not found exported service: com.zhuanche.driver.service.DriverInfoService:8080 in [com.zhuanche.driver.service.DriverDutyService:8080, com.zhuanche.driver.service.driver.DriverJoinRecordService:8080], may be version or group mismatch , channel: consumer: /10.30.3.60:48174 --> provider: /10.30.3.72:8080, message:RpcInvocation [methodName=findByPhone, parameterTypes=[class java.lang.String], arguments=[13826557857], attachments={path=com.zhuanche.driver.service.DriverInfoService, input=256, dubbo=2.5.8, interface=com.zhuanche.driver.service.DriverInfoService, version=0.0.0, timeout=5000}]
com.alibaba.dubbo.remoting.RemotingException: Not found exported service: com.zhuanche.driver.service.DriverInfoService:8080 in [com.zhuanche.driver.service.DriverDutyService:8080, com.zhuanche.driver.service.driver.DriverJoinRecordService:8080], may be version or group mismatch , channel: consumer: /10.30.3.60:48174 --> provider: /10.30.3.72:8080, message:RpcInvocation [methodName=findByPhone, parameterTypes=[class java.lang.String], arguments=[13826557857], attachments={path=com.zhuanche.driver.service.DriverInfoService, input=256, dubbo=2.5.8, interface=com.zhuanche.driver.service.DriverInfoService, version=0.0.0, timeout=5000}]
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol.getInvoker(DubboProtocol.java:205)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:76)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:98)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)
at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:81)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

　　   在某一個時刻，大量的dubbo 異常。 早上起來先去elk里面查詢是不是並發量的問題，查了下這個時刻 並發量每秒大概300-400的量，和平時差別不大。然后找到具體的接口，通過elk的 request_time 發現有個別接口響應時間達到 1秒多,而調用的dubbo服務

   @Reference(timeout = 1000)

private DriverInfoService driverInfoService;

 時間超過了1秒。但是為什么會報錯那。。。 正常來說又不會出現問題


 看了 消費者，報的的dubbo 異常，之后再查看生產者，查了下這個時間段日志是正常的。 但是最后查看dubbo的啟動日志時候：
 

03:07:23,213 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - RocketmqRemoting level set to INFO
03:07:23,213 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [RocketmqClientAppender] to Logger[RocketmqRemoting]
03:07:23,213 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [RocketmqClient] to false
03:07:23,213 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - RocketmqClient level set to INFO
03:07:23,213 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [RocketmqClientAppender] to Logger[RocketmqClient]
03:07:23,213 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
03:07:23,213 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@4b87074a - Registering current configuration as safe fallback point
[2019-07-18 03:07:28] Dubbo service server started!

　　

 

 

  dubbo服務 在這個時間段內竟然重啟了。。。

   之后聯系運維，原來是他們在凌晨這個時間點 重啟了swap top5的服務。剛好這個時間段請求量比較大，zk注冊的時候還沒有注冊上。導致請求了重啟的那個機器ip上面。從而出現了大批量的報錯信息。