早上上班后得知，服務費未同步到代理商系統。查看draft_server系統生產環境的log，顯示在往RabbitMQ推數據時出現異常：no route to host。
2019-07-29 01:30:00,136 INFO [pool-13-thread-30] 201154611 (AgentProfitProducer.java:32) - 代理商服務費入隊
2019-07-29 01:31:01,713 INFO [org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer#0-2021] 201216188 (SimpleMessageListenerContainer.java:1453) - Restarting Consumer: tags=[{}], channel=null, acknowledgeMode=AUTO local queue size=0
2019-07-29 01:31:02,150 INFO [pool-13-thread-30] 201216625 (AgentProfitServiceImpl.java:182) - [代理商服務費推送]-異常
org.springframework.amqp.AmqpIOException: java.net.NoRouteToHostException: No route to host (Host unreachable)
at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:71) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:309) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:547) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createConnection(ConnectionFactoryUtils.java:90) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:140) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:76) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1374) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1367) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:699) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
--
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_191]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_191]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_191]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_191]
at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_191]
at com.rabbitmq.client.impl.FrameHandlerFactory.create(FrameHandlerFactory.java:32) ~[amqp-client-3.6.3.jar:?]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:811) ~[amqp-client-3.6.3.jar:?]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:725) ~[amqp-client-3.6.3.jar:?]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:296) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
... 21 more
2019-07-29 01:31:02,150 INFO [pool-13-thread-30] 201216625 (AgentProfitServiceImpl.java:184) - 代理商服務費推送結束2019-07-29T01:31:02.150+0800
打開vpn連接到生產環境，用本地test程序嘗試往生產的mq推數據，發現正常。接下來，rpc調用生產的服務費推送服務，再看生產log，mq依然有問題。不過這次是SocketTimeoutException。
2019-07-29 13:57:23,514 INFO [pool-13-thread-38] 245997989 (AgentProfitProducer.java:32) - 代理商服務費入隊
2019-07-29 13:57:47,563 WARN [org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer#0-2621] 246022038 (SimpleMessageListenerContainer.java:1462) - Consumer raised exception, processing can restartif the connection factory supports it
2019-07-29 13:57:47,564 INFO [org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer#0-2621] 246022039 (SimpleMessageListenerContainer.java:1453) - Restarting Consumer: tags=[{}], channel=null, acknowledgeMode=AUTO local queue size=0
2019-07-29 14:00:23,636 INFO [pool-13-thread-38] 246178111 (AgentProfitServiceImpl.java:182) - [代理商服務費推送]-異常
org.springframework.amqp.AmqpIOException: java.net.SocketTimeoutException: connect timed out
at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:71) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:309) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:547) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createConnection(ConnectionFactoryUtils.java:90) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:140) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:76) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1374) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1367) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:699) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
--
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_191]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_191]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_191]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_191]
at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_191]
at com.rabbitmq.client.impl.FrameHandlerFactory.create(FrameHandlerFactory.java:32) ~[amqp-client-3.6.3.jar:?]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:811) ~[amqp-client-3.6.3.jar:?]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:725) ~[amqp-client-3.6.3.jar:?]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:296) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
... 21 more
2019-07-29 14:00:23,636 INFO [pool-13-thread-38] 246178111 (AgentProfitServiceImpl.java:184) - 代理商服務費推送結束2019-07-29T14:00:23.636+0800
2019-07-29 14:00:47,648 WARN [org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer#0-2622] 246202123 (SimpleMessageListenerContainer.java:1462) - Consumer raised exception, processing can restartif the connection factory supports it
org.springframework.amqp.AmqpIOException: java.net.SocketTimeoutException: connect timed out
at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:71) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:309) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:547) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createConnection(ConnectionFactoryUtils.java:90) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:140) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:76) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:472) ~[spring-rabbit-1.6.1.RELEASE.jar:?]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1280) [spring-rabbit-1.6.1.RELEASE.jar:?]
繼續分析log，奇怪地發現在這兩次往mq放數據之前，都有一個奇怪的Restarting Consumer。
draft_server不僅是mq生產者，還是mq消費者。登陸rabbitmq管理控制台，隊列顯示的竟然是... no consumers ...。那么，問題也許出現在這里。服務啟動后應該自動注冊的，看來上周五上線未正常發版（人為手動刪掉consumer或手工創建隊列的幾率不大）。
於是，申請讓運維同事重新發版，Jenkins構建完畢，服務重啟，發現隊列有消費者了。
然后，本地再次rpc調用服務器上的那個服務，一切正常，mq可以正常生產消息了。