今天遇到一個Eureka集群的一個坑。
問題現場類似是這樣的：兩台Eureka組成的服務注冊中心集群，兩台服務提供方server1、server2，兩個服務調用方client1、client2。
正常的情況下：
client1和client2通過服務中心獲取的服務提供方的注冊的元數據信息，然后本地緩存調用server2、server1的服務API是沒問題的。調用server的api服務時采用的FeignClient通過eureka注冊中心獲取服務信息調用。
以下簡寫成client服務、server服務、eureka服務：
但今天遇到的情況是client調用server時，返回結果不穩定，以下是排查過程。
1、習慣性的先查看client方日志，發現接口調用有一定的概率失敗，但是並沒有拋異常，而僅僅是response返回null。
2、查看server方日志，並沒有發現異常，也沒有發現response為null的情況，並且server方的本機上訪問接口都是響應正常。
3、通過分析兩方日志，猜想client端的請求可能沒有打到server方。
4、分析整個調用流程，只差eureka服務這一個環節沒有排查了，然后查看eureka服務上的server方注冊信息是否正常。
5、發現兩台eureka服務，有一台tomcat假死了，至此猜想是假死的eureka服務假死所致。
6、驗證猜想，重啟假死的eureka的tomcat服務，問題驗證解決。
那么問題來了，
1、為什么一台eureka假死了，沒有拋異常，只在response中返回個null?
2、eureka集群的一台實例假死，為什么會導致服務間調用不穩定？
3、eureka服務跑了大半年，怎么突然一台實例假死了？
接下來，我們來探討這三個問題。
1、為什么一台eureka假死了，沒有拋異常，只在response中返回個null?
這個問題經查，eureka假死，導致調用方調用服務方時，觸發了hystrix熔斷器,項目實現的熔斷降級接口直接在熔斷自動降級時返回null，這就回答了第一個問題。
2、eureka集群的一台實例假死，為什么會導致服務間調用不穩定？
這個得翻看一下eureka的機制源碼了，猜想feign端在本地緩存90s后去刷新本地服務列表信息時，卡在調用eureka服務上了（此時恰好通過假死的eureka服務刷新），從而服務不可用。
3、eureka服務跑了大半年，怎么突然一台實例假死了？
生產環境有多台虛擬機eureka服務器，配置都一樣，其它的服務器沒有出過問題，只有一台eureka服務會有類似周期性假死的問題，由於筆者生產環境權限所限，只能猜想可能是宿主機的原因，估計換台機器可能就沒問題了，待驗證。
還有幾個等思考的問題，之后待續。。。。。。。。。。。。。
1、服務提供方配置了多個eureka 地址，是服務注冊時，是調用注冊到哪個上？ ---> 猜想優化使用第一個，那第二個是何時用到的（除了第一個宕機的情況）
2、feign接口通過eureka調用服務時，是不需要自己項目也注冊到eureka上的，怎么處理？
3、服務調用方配置了多個eureka 地址，是獲取服務列表調用時，是調用注冊到哪個上？ --> 猜想優化使用第一個，那第二個是何時用到的（除了第一個宕機的情況）
另外，昨天查eureka假死日志時，日志打的異常堆棧如下：
2019-06-13 18:07:58.907 INFO 23945 --- [a-EvictionTimer] c.n.e.registry.AbstractInstanceRegistry : Running the evict task with compensationTime 0ms
2019-06-13 18:08:04.949 ERROR 23945 --- [nio-8001-exec-5] o.s.boot.web.support.ErrorPageFilter : Forwarding to error page from request [/] due to exception [Filter execution threw an exception]
javax.servlet.ServletException: Filter execution threw an exception
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:255) [catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:110) [spring-boot-1.4.3.RELEASE.jar:1.4.3.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:719) ~[catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:465) ~[catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:390) ~[catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:317) ~[catalina.jar:8.0.30]
at org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel(InternalResourceView.java:168) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:303) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1257) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1037) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:980) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:622) ~[servlet-api.jar:na]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) ~[servlet-api.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291) [catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-websocket.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) ~[spring-boot-1.4.3.RELEASE.jar:1.4.3.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105) ~[spring-boot-actuator-1.4.3.RELEASE.jar:1.4.3.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89) ~[spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) ~[spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) ~[spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106) ~[spring-boot-actuator-1.4.3.RELEASE.jar:1.4.3.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:117) [spring-boot-1.4.3.RELEASE.jar:1.4.3.RELEASE]
at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:61) [spring-boot-1.4.3.RELEASE.jar:1.4.3.RELEASE]
at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:92) [spring-boot-1.4.3.RELEASE.jar:1.4.3.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:110) [spring-boot-1.4.3.RELEASE.jar:1.4.3.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.30]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) [catalina.jar:8.0.30]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) [catalina.jar:8.0.30]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) [catalina.jar:8.0.30]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) [catalina.jar:8.0.30]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [catalina.jar:8.0.30]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) [catalina.jar:8.0.30]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521) [catalina.jar:8.0.30]
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096) [tomcat-coyote.jar:8.0.30]
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674) [tomcat-coyote.jar:8.0.30]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500) [tomcat-coyote.jar:8.0.30]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456) [tomcat-coyote.jar:8.0.30]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_20]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_20]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.0.30]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_20]
Caused by: java.lang.NoClassDefFoundError: com/thoughtworks/xstream/io/xml/PrettyPrintWriter
at com.thoughtworks.xstream.io.xml.DomDriver.createWriter(DomDriver.java:122) ~[xstream-1.4.9.jar:1.4.9]
at com.thoughtworks.xstream.io.xml.DomDriver.createWriter(DomDriver.java:127) ~[xstream-1.4.9.jar:1.4.9]
at com.thoughtworks.xstream.XStream.toXML(XStream.java:1042) ~[xstream-1.4.9.jar:1.4.9]
at com.netflix.discovery.converters.wrappers.CodecWrappers$XStreamXml.encode(CodecWrappers.java:377) ~[eureka-client-1.4.12.jar:1.4.12]
at com.netflix.discovery.provider.DiscoveryJerseyProvider.writeTo(DiscoveryJerseyProvider.java:135) ~[eureka-client-1.4.12.jar:1.4.12]
at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:302) ~[jersey-server-1.19.1.jar:1.19.1]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510) ~[jersey-server-1.19.1.jar:1.19.1]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) ~[jersey-server-1.19.1.jar:1.19.1]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) ~[jersey-server-1.19.1.jar:1.19.1]
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) ~[jersey-servlet-1.19.1.jar:1.19.1]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) ~[jersey-servlet-1.19.1.jar:1.19.1]
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:927) ~[jersey-servlet-1.19.1.jar:1.19.1]
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875) ~[jersey-servlet-1.19.1.jar:1.19.1]
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829) ~[jersey-servlet-1.19.1.jar:1.19.1]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.30]
... 93 common frames omitted
2019-06-13 18:08:58.907 INFO 23945 --- [a-EvictionTimer] c.n.e.registry.AbstractInstanceRegistry : Running the evict task with compensationTime 0ms
2019-06-13 18:09:42.160 INFO 23945 --- [hresholdUpdater] c.n.e.r.PeerAwareInstanceRegistryImpl : Current renewal threshold is : 102
2019-06-13 18:09:58.907 INFO 23945 --- [a-EvictionTimer] c.n.e.registry.AbstractInstanceRegistry : Running the evict task with compensationTime 0ms
2019-06-13 18:10:19.760 INFO 23945 --- [trap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver : Resolving eureka endpoints via configuration
這個其實就是找不到報錯了找不到相應的url所致，可以設置com.ibm.ws.webcontainer.invokeFlushAfterService為false，關閉此日志。
網上也有說可以增加如下依賴解決關閉日志，沒試過，此處只記錄一下。
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-thymeleaf</artifactId>
<version>${spring.boot.version}</version>
</dependency>
<!--HTML掃描器和標簽補償器-->
<dependency>
<groupId>net.sourceforge.nekohtml</groupId>
<artifactId>nekohtml</artifactId>
<version>1.9.22</version>
</dependency>