java應用掛死故障排查

現象：

　　java開發的web應用無法訪問

排查：

1．從resin/log/watchdog-manager.log的日志里可以看出來，jvm的內存滿，無法創建新進程

java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:574)
        at com.caucho.util.ThreadPool$ThreadLauncher.startConnection(ThreadPool.java:812)
        at com.caucho.util.ThreadPool$ThreadLauncher.run(ThreadPool.java:851)
        at java.lang.Thread.run(Thread.java:595)

　　

2．用jstat –gcutil pid可以看出jvm的OLD堆內存區域已經100%，與上述判斷符合

S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT  

0.00   0.00  99.19 100.00  20.72    577   62.517   322 2953.102 3015.619

　　

3．用jstack pid查看jvm的內部的執行線程情況，可以看到存在大量的下述線程，阻塞

Thread 15810: (state = BLOCKED)

 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park() @bci=5, line=118 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=41, line=1767 (Interpreted frame)
 - com.ctsi.nmp.flux.ecfchart.ECFluxChartService.writeChart(com.ctsi.nmp.flux.ecfchart.ECFluxChartType, com.ctsi.nmp.flux.ecfchart.E

CFluxType, java.lang.String, java.util.Date, java.util.Date, java.util.List, boolean, java.io.OutputStream) @bci=295, line=189 (Compiled frame)

 - com.ctsi.nmp.flux.ecfchart.ECFluxChartClient.executeChart() @bci=1051, line=203 (Compiled frame)
 - sun.reflect.GeneratedMethodAccessor49.invoke(java.lang.Object, java.lang.Object[]) @bci=36 (Compiled frame)
 - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=25 (Compiled frame)
 - com.ctsi.nmp.flux.ecfchart.ECFluxChartClient.execute() @bci=94, line=230 (Interpreted frame)
 - _jsp._nos._flux._flux._chart._ECFluxChart__jsp._jspService(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletR
esponse) @bci=130, line=39 (Compiled frame)
 - com.caucho.jsp.JavaPage.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=9, line=61 (Interpreted frame)
 - com.caucho.jsp.Page.pageservice(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=252, line=578

 (Compiled frame)
 - com.caucho.server.dispatch.PageFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=262, line=195 (Compiled frame)
 - com.zhongying.util.EncodingFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain
) @bci=39, line=80 (Compiled frame)
 - com.caucho.server.webapp.WebAppFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=140, line=187 (Compiled frame)
 - com.caucho.server.dispatch.ServletInvocation.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=97, line=266 (Compiled frame)
 - com.caucho.server.hmux.HmuxRequest.handleRequest() @bci=478, line=435 (Compiled frame)
 - com.caucho.server.port.TcpConnection.run() @bci=375, line=678 (Compiled frame)
 - com.caucho.util.ThreadPool$Item.runTasks() @bci=246, line=721 (Compiled frame)
 - com.caucho.util.ThreadPool$Item.run() @bci=106, line=643 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=595 (Interpreted frame)

　　

 可以判斷是應用的com.ctsi.nmp.flux.ecfchart.ECFluxChartClient.executeChart部分出現了問題，導致大量的線程阻塞，研發優化這部分代碼后問題未再重現