記一次使用OkHttpClient大量請求導致SocketTimeoutException再到OutOfMemoryError的問題

需求：因為做一個pc資訊站點，服務端需要主動緩存，所以寫了一個調度每10分鍾更新熱點數據的頁面緩存

場景：本地/開發/測試 環境的時候，因為主動緩存的html文件比較少，沒有發現這個問題，功能上線后看到項目啟動正常，測試通過后，就沒管正式版服務器了，沒想到凌晨的時候服務器宕機，提示OOM異常

bug復現：線上需要緩存的文件上萬條，調度每次使用單線程的方式，主動發出請求內網訪問我們的站點，觸發生成更新緩存的動作

OOM之前出現了大量的連接超時

ERROR 2020-06-01 09:44:13,918 [schedule-pool-3] c.x.u.h.MyHttpClient.get 142 :java.net.SocketTimeoutException: timeout

大量連接超時后就出現了OOM

ERROR 2020-06-01 09:51:52,527 [catalina-exec-18] o.s.b.w.s.s.ErrorPageFilter.forwardToErrorPage 183 :Forwarding to error page from request [/videos/short/0123168.html] due to exception [unable to create new native thread]
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:717)
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
        at okhttp3.ConnectionPool.put(ConnectionPool.java:135)
        at okhttp3.OkHttpClient$1.put(OkHttpClient.java:106)
        at okhttp3.internal.http.StreamAllocation.findConnection(StreamAllocation.java:182)
        at okhttp3.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:123)
        at okhttp3.internal.http.StreamAllocation.newStream(StreamAllocation.java:93)
        at okhttp3.internal.http.HttpEngine.connect(HttpEngine.java:296)
        at okhttp3.internal.http.HttpEngine.sendRequest(HttpEngine.java:248)
        at okhttp3.RealCall.getResponse(RealCall.java:243)
        at okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:201)
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:163)
        at okhttp3.RealCall.execute(RealCall.java:57)
        at com.xmz1.util.http.MyHttpClient.get(MyHttpClient.java:113)
        at com.xmz1.util.request.RequestUtil.getData(RequestUtil.java:143)
        at com.xmz1.util.request.RequestUtil.getDTO(RequestUtil.java:55)
        at com.qiuzhang.jserver.mobile.service.video.impl.ShortVideoServiceImpl.getDetail(ShortVideoServiceImpl.java:69)
        at com.qiuzhang.jserver.mobile.controller.video.VideoController.getDetail(VideoController.java:217)
        at sun.reflect.GeneratedMethodAccessor187.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:893)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:798)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.qiuzhang.jserver.mobile.conf.LogCostFilter.doFilter(LogCostFilter.java:43)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)
        at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
        at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
        at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:94)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:128)
        at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:66)
        at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:103)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:121)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

1、服務器配置上jmx參數，通過jconsole排查

配置了2台tomcat，其中項目啟動后可以發現線程數和堆空間一直飆升，其中一台的數據可以感覺出線程數5分鍾下降一次

一開始以為是連接太多導致所以修改了連接超時時間（connectTimeout、readTimeout）

結果發上去觀察一段時間后還是沒用

於是只好上網找答案，發現網上有人說是OkHttpClien源碼中設定keepAlive時間為5分鍾，瞬間恍然大悟

晚上給的解決方案：

在短時間內發起了大量網絡連接，每個是一個線程，而且每個都默認保存5分鍾，很快線程數就超標了。
考慮到我的每次請求都只是主動觸發更新緩存動作，所以我修改了ConnectionPool的keepAliveDuration時間，讓每次連接10后就關閉。

 

修改后發（11點發布），ok，解決問題！線程數不會在狂飆了，堆空間還要繼續排查一下

 

堆空間排插

使用 java 自帶的 jvisualvm

 

 

 

 使用 jconsle 的配置就可以直接連接上了，可以看出來堆空間一直居高不下

 

我們導出堆文件，等待一段時間后，從遠程服務器上導入到 visualVM 

 

 

 

 點擊第一個實例數量最多的可以看到，是我們頁面緩存數據，ok 找到問題，然后就可以解決了，

這里我的解決方法是重寫了ConcurrentLRUHashMap 方法限制了 map 的最大值

 

 

 

 

 

 

 

 

參考資料：https://blog.csdn.net/tianyaleixiaowu/article/details/78811488